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The  Military  Operations  Research  Society 

The  purpose  of  the  Military  Operations  Research  Society  (MORS)  is  to  enhance  the  quality  and 
effectiveness  of  classified  and  unclassified  military  operations  research.  To  accomplish  this  purpose,  the 
Society  provides  media  for  professional  exchange  and  peer  criticism  among  students,  theoreticians, 
practitioners,  and  users  of  military  operations  research.  These  media  consist  primarily  of  the  traditional 
annual  MORS  symposia  (classified),  their  published  abstracts/proceedings,  special  mini-symposia, 
workshops,  colloquia  and  special  purpose  monographs.  The  forum  provided  by  these  media  is  directed  to 
display  the  state  of  the  art,  to  encourage  consistent  professional  quality,  to  stimulate  communication  and 
interaction  between  practitioners  and  users,  and  to  foster  the  interest  and  development  of  students  of 
operations  research.  In  performing  its  function,  the  Military  Operations  Research  Society  does  not  make  or 
advocate  official  policy  nor  does  it  attempt  to  influence  the  formulation  of  policy.  Matters  discussed  or 
statements  made  during  the  course  of  its  symposia  or  printed  in  its  publications  represent  the  positions  of 
the  individual  participants  and  authors  and  not  of  the  Society. 

The  Military  Operations  Research  Society  is  governed  by  a  Board  of  Directors  consisting  of  30  members, 
28  of  whom  are  elected  by  vote  of  the  Board  to  serve  a  term  of  four  years.  The  persons  nominated  for  this 
election  are  normally  individuals  who  have  attained  recognition  and  prominence  in  the  field  of  military 
operations  research  and  who  have  demonstrated  an  active  interest  in  its  programs  and  activities.  The 
remaining  two  members  of  the  Board  of  Directors  are  the  immediate  Past  President  who  serves  by  right 
and  the  Executive  Vice  President  who  serves  as  a  consequence  of  his  position.  A  limited  number  of 
Advisory  Directors  are  appointed  from  time  to  time,  usually  for  a  one-year  term,  to  perform  some 
particular  function.  Since  a  major  portion  of  the  Society's  affairs  is  connected  with  classified  services  to 
military  sponsors,  the  Society  does  not  have  a  general  membership  in  the  sense  of  other  professional 
societies.  The  members  of  MORS  are  the  Directors,  persons  who  have  attended  a  MORS  meeting  within 
the  past  three  years  and  Fellows  of  the  Society  (FS)  who,  in  recognition  of  their  unique  contributions  to  the 
Society,  are  elected  by  the  Board  of  Directors  for  life. 

MORS  is  sponsored  by: 

•  The  Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

•  The  Director,  Assessment  Division,  Office  of  the  Chief  of  Naval  Operations 

•  The  Director  of  Command  and  Control,  Deputy  Chief  of  Staff,  Air  and  Space  Operations,  US  Air 
Force 

•  The  Commanding  General,  Marine  Corps  Combat  Development  Command 

•  The  Director  of  Force  Structure,  Resource  and  Assessment,  The  Joint  Staff 

•  The  Director  Program  Analysis  and  Evaluation,  Office  Secretary  of  Defense 
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MORS  Advanced  Distributed  Simulations  for  Analysis  ’96 

Workshop 
Executive  Summary 


1.  Introduction.  Advanced  Distributed 
Simulations  (ADS)  and  their  appropriate  use 
by  the  defense  analytical  conununity  were 
the  focus  of  a  MORS  Workshop,  ADSA  '96, 
sponsored  by  ODUSA(OR),  OCNO(N81), 
HQ  USAF/XOM,  Joint  Staff  (J8)  and  OSD 
(PA&E).  ADSA  '96  was  another  in  a  long 
series  of  MORS  Workshops  whereby 
MORS  leadership  provides  a  forum  for 
experienced  military  analysts  from  all 
Services  and  the  supporting  civilian 
communities  to  weigh  in  on  a  tough 
question  or  issue  facing  the  defense 
community.  Generally,  these  workshops 
take  the  form  of  working  sessions  where 
participants  strive  to  clarify  the  question  or 
issue  and  propose  approaches  to  assist  the 
DoD  in  issue  resolution.  These  forums  have 
served  the  defense  community  well  by 
providing  insight  as  to  where  DoD  anal5rtical 
efforts  should  be  focused  or  leveraged,  or,  in 
some  c£ises,  to  help  the  analytical 
community  better  understand  major  changes 
in  policy  affecting  their  work. 

For  this  particular  Workshop,  over  100 
senior  military  and  civilian  analysts  gathered 
in  Williamsburg,  VA,  from  30  January  to  1 
February  1996  to  have  a  frank  discussion  on 
the  applicability/utility  of  ADS  and 
Distributed  Interactive  Simulation  (DIS) 
technologies  to  the  analytical  community 
and  to  document  findings  and 
reconmiendations  resulting  from  the 
discussion.  Much  of  the  discourse  that 
preceded  the  Workshop  and  continued 
through  the  initial  Workshop  sessions 
focused  on  the  need  for  a  definition  for 
ADS.  Ed  Brady,  FS,  the  General  Chair, 
believed  that  any  definition  would  restrict 
the  potential  applicability  of  ADS  but  finally 
proffered  the  following; 


Advanced  Distributed  Simulation  —  The 
evolving  DoD  distributed  modeling  and 
simulation  (M&S)  infrastructure, 
including  synthetic  environments,  run¬ 
time  infrastructures  and  connected 
human-in-the  loop  (HITL)  simulations 
such  as  DIS. 

For  the  purposes  of  this  discourse,  both  ADS 
and  DIS  are  distributed  but  ADS  is 
considered  a  much  broader  set  of  modem 
M&S  capabilities  than  DIS.  For  example, 
ADS  has  the  potential  to  include  the 
interaction  of  constractive-to-constmctive 
simulations  and  constructive-to-live 
simulations  as  well  as  the  virtual  simulations 
connected  by  DIS.  Note  also  that  this 
definition  does  not  require  an  HITL  as  part 
of  any  particular  instance  of  an  ADS.  The 
variability  of  humans-in-the  loop  is  a  major 
concern  to  analysts  dependent  upon 
replicability  of  experiments 

Leveraging  off  lessons  learned  from 
previous  MORS  Workshops,  advance 
preparation  for  the  ADSA  '96  included  a 
facilitators  training  session  for  chairs  and 
co-chairs  in  Arlington,  VA  —  scheduled  in 
between  the  multiple  snow  storms  that 
blanketed  Washington  most  of  that  winter. 
This  training  session  was  to  acquaint  the 
chairs  (listed  in  Table  1)  and  co-chairs  with 
known  ADS  issues  and  to  prepare  them  for 
getting  the  most  out  of  their  respective 
working  groups.  In  addition,  all  Workshop 
participants  were  provided  a  healthy  reading 
package  (see  Table  2)  in  advance  of  the 
actual  Workshop. 
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2.  Chair's  Charge.  During  his  comments  at 
the  opening  of  ADSA  '96,  Ed  Brady,  FS 
emphasized  to  the  Workshop  participants 
that  they  should  focus  on  identifying 
potential  solutions  that  would  be  of  value  to 
both  the  analytical  community  and  the  M&S 
community.  In  particular,  the  participants 
were  asked  to  identify: 

(1)  Appropriate  uses  of  ADS/DIS 
capabilities  by  the  analytical 
community 


(2)  Inherent  limitations  and 
advantages  of  ADS/DIS  capabilities 

(3)  Current  shortcomings  of 
ADS/DIS  capabilities,  that  if 
remedied  could  significantly 
improve  the  utility  of  ADS/DIS 
capabilities  to  the  analytical 
community 

(4)  Ways  to  analyze  and  test  the 
relevance  and  quality  of  ADS/DIS- 
based  tools 

for  the  analytical  community 


Table  1:  ADSA  '96  Workshop  Chairs  and  Working  Group  Chairs 


General  Chair  -  ADSA  '96 

Ed  Brady,  FS 

Strategic  Perspectives  Inc. 

Working  Group  I-  Battlefield 
Effectiveness  Chair 

Mike  Bauman 

Army  HQ  TRAC 

Working  Group  2  -  Material, 
Systems  and  Acquisition 

Chair 

LtCol  Bob  Sheldon 

Air  Force  AFS  AA/SAZ 

Working  Group  3  -  Test  and 
Evaluation 

Chair 

Dr.  Flank  Dubin 

Army  OPTEC 

Working  Group  4  -  Training, 
Mission  Rehearsal  and  Alternate 
Courses  of  Action 

Chair 

Fred  Hartman 

Foxhall  Group 

Working  Group  5  -  Analysis  and 
Requirements 

Chair 

Dr.  Cy  Staniec 

OSD(PA&E) 

Working  Group  6  -  Understanding 
Behavior/Performance 

Chair 

Dr.  John  Friel 

RAND 

Working  Group  7  -  Synthesis  Chair 

Dr.  Stu  Starr 

MITRE 

Technical  Co-Chairs 

Ted  Bean  and  Tana  Reagan 

MITRE 

2 


Table  2;  ADS  A  '96  Read  Ahead  Package  for  Participants 


Sikora,  Jim,  and  Phil  Coose 

"What  in  the  World  is  ADS? 
PHALANX  June  1995 

Under  Secretary  of  Defense  for 
Acquisition  and  Technology 

Modeling  and  Simulation  (M&S) 
Master  Plan 

DMSO 

HLA  Management  Plan  High  Level 
Architecture  of  Modeling  and 
Simulation,  Version  1. 6 

Davis,  Paul  K 

Distributed  Interactive  Simulation 
in  the  Evolution  ofDoD  Warfare 

Davis,  P.  K.,  and  D.  Blumenthal 

The  Base  of  Sand  Problem:  A 
White  Paper  on  the  State  of 
Military  Comhat  Modeling 

Hammer,  Peter  L. 

Annals  of  Operations  Research 

Lese,  Jr.,  William  G.  and  Jim 
Metzger 

Future  Joint  Warfare  Analysis 
Model  Designs 

Oswalt,  Ivar 

Technology  Trends  in  Military 
Simulation 

Smith,  Mark  E. 

Determining  the  Utility  of 
Advanced  Distributed  Simulation  to 
Test  and  Evaluation 

Kloeber,  Jr.,  Jack  M.  and  Jack  A. 
Jackson 

Issues  in  Using  a  DIS  Facility  in 
Analysis 

Bolling,  Robert  H. 

The  Joint  Theater  Level  Simulation 
in  Military  Operations  Other  than 
War 

MORS  Mini-Symposium 

Distributed  Interactive  Simulation 
(DIS)  Proceedings 

Products  resulting  from  the  Workshop  were 
to  be  focused  on  the  clarification  of  issues 
and  problems  and  the  identification  of 
potential  solution  "spaces."  In  particular,  the 
expectations  of  MORS  Sponsors  included 
understanding  ways  to  use  ADS -based 
analysis  that  differed  from  those  of  other 
analyses  as  well  as  identification  of  classes  of 
analyses  most  amenable  to  support  by 
ADS-based  tools.  Those  classes  of  analyses 
for  which  ADS  tools  are  not  useful  would 
also  be  very  useful  to  MORS  Sponsors.  This 
required  the  participants  to  think  back  on  the 
question:  "Why  do  we  analyze  what  we 
analyze?"  Is  it  because  of  available  analysis 
tools?  Our  own  limited  knowledge?  How  can 
ADS-based  tools  be  used  to  give  us  more 
insight?  More  importantly,  given  the 
changing  nature  of  the  questions  military 
analysts  are  being  required  to  answer,  what 
class  of  questions  looks  amenable  to  analysis 
by  ADS  based  tools  independent  of  whether 
or  not  they  are  currently  available  or 
technologically  feasible?  Clearly,  an  answer 
to  this  last  question  would  help  focus 
ongoing  research.  Another  area  of  particular 
interest  is  the  identification  of  experiments 
designed  to  assist  in  the  development  of 
Measures  of  Merit  (MoMs)  for  analysis  of 
Command  and  Control  (C2)  and  other 
heavily  human-related  functions.  Given  the 
evolving  nature  of  ADS  and  the  changing 
questions  being  put  to  analysts,  there  was 
room  in  the  participants’  discussions  for 
many  differing  views  of  ADS  and  areas  ripe 
for  its  potential  use. 

The  remaining  ADS  A  '96  Workshop  format 
consisted  of  (1)  an  overview  of  ADS 
Technology  by  Dr.  Randy  Garrett  of 
DARPA,  (2)  a  keynote  address  by  Dr.  Anita 
K.  Jones,  Director  of  Defense  Research  and 
Engineering,  OSD,  (3)  a  senior  practitioner's 
panel  and  (4)  the  kickoff  of  working  group 
sessions.  The  Working  Group  (WG) 
sessions  continued  through  the  last  day  with 


WG  Chairs  reporting  out  the  results  of  their 
sessions  the  last  afternoon 

3.  Keynote  Address.  A  summary  of  Dr. 
Jones'  ke)mote  address  is  available  in  the 
June  1996  PHALANX.  For  our  purposes. 
Dr.  Jones  observed  that  it  was  propitious  to 
conduct  an  ADS  workshop  since  there  are 
several  things  that  are  new  in  the  world  of 
the  analyst. 

First,  the  analyst  is  being  asked  to  address 
questions,  some  of  which  are  new,  in  a  more 
sophisticated  way.  With  the  dissolution  of 
the  Warsaw  Pact  and  the  Soviet  Union,  the 
world  is  proving  to  be  much  more  complex 
from  a  national  security  perspective.  For 
example,  the  U.S.  faces  a  much  more 
ambiguous  threat  from  multiple  countries 
with  diverse  capabilities;  defense  planning 
revolves  around  "multiple  scenario" 
problems;  and  the  U.S.  military  and  its  allies 
are  being  asked  to  undertake  new  types  of 
military  operations,  such  as  complex 
humanitarian  operations  in  Africa  and 
peacekeeping  in  the  former  Yugoslavia. 
Since  most  of  these  issues  involve  complex 
joint/  coalition  activities,  the  analyst  must 
consider  the  effect  of  joint/coalition 
involvement  at  the  very  outset  of  the 
analysis.  In  addition,  it  was  not  unusual, 
several  years  ago,  for  analysts  to  assume 
"perfect"  C2  and  to  perform  assessments 
within  that  context.  In  view  of  the 
importance  of  C2  in  the  Chairman  of  the 
Joint  Chiefs  of  Staff  (CJCS)  Vision  2010, 
such  an  assumption  is  clearly  no  longer 
acceptable.  Finally,  there  is  increasing 
recognition  of  the  importance  that  a  broad 
set  of  environmental  effects  can  have  on 
analysis  (for  example,  more  granular  terrain 
to  capture  the  effects  of  terrain  masking;  the 
properties  of  obscurants  and  their  effects  on 
the  performance  of  intelligence,  surveillance 
and  reconnaissance  (ISR)  systems;  and  the 
challenges  posed  by  detecting. 


4 


characterizing  and  reacting  to  chemical  and 
biological  agents). 

Second,  the  analyst  is  being  asked  to 
communicate  information  more  effectively 
to  decision  makers.  Most  decision  makers 
are  no  longer  satisfied  with  columns  of 
numbers.  They  want  visualization  and 
animation  tools  to  capture  the  salient  aspects 
of  an  analysis  and  the  opportunity  to 
participate  in  a  dynamic,  "what  if..." 
dialogue  with  the  analyst. 

Finally,  it  was  observed  that  several 
communities  (e.g.,  trainers,  acquirers  of 
systems)  are  acquiring  new  simulations  and 
data,  both  of  which  have  the  potential  to 
support  analysts  substantively.  This  suggests 
the  desirability  of  working  more  closely 
with  these  communities,  both  to  leverage 
their  investments  and  to  share  the 
simulations  and  data  developed  by  the 
analytical  community.  The  DoD  "high  level 
architecture"  initiative  was  also  introduced 
as  providing  an  evolvable  and  accessible 
framework  with  a  common  support 
structure,  common  rules  for  how 
components  interact,  and  shared  support 
services  to  support  the  front-end  analysis 
that  must  be  done  as  part  of  any  simulation 
design  process.  The  structure  and 
commonality  encourage  interoperability  of 
simulations. 

4.  Senior  Practitioner  Panel.  As  Ed  Brady, 
FS  introduced  the  senior  practitioner  panel, 
he  noted  that  it  was  placed  on  the  Workshop 
agenda  to  motivate  the  participants  with  the 
diversity  of  opinion  on  ADS.  Although 
working  groups  were  formed  around  a  major 
area,  they  were  not  independent  and  needed 
to  be  aware  of  other  organizational  and 
individual  opinions  as  they  went  through 
their  own  deliberations.  The  comments  of 
each  of  these  panel  members  are 
summarized  below. 


Dr.  Randy  Garrett  (DARPA)  —  Dr. 
Garrett  highlighted  the  issue  of  technology 
transfer  in  relationship  to  acquisition, 
stressing  that  we  must  find  quicker  ways  to 
take  innovative  solutions  out  of  the  labs  and 
into  systems  where  they  can  gain  wider 
acceptance  and  usage.  He  asked  the  working 
groups  to  consider  the  obstacles  that  keep 
the  two  communities  from  sharing.  On  the 
high  level  architecture,  he  observed  that  the 
city  planning  analogy  as  a  technology 
framework  for  achieving  interactions  among 
simulations  and  live  C2  systems,  only  goes 
so  far.  Building  codes  used  by  city  planners 
are  based  upon  a  lot  of  experience;  civil 
engineers  have  a  long  history  of  knowing 
what  works  and  what  doesn’t.  But  we  are 
still  learning  in  the  software  development 
community.  We  have  become  pretty  good  in 
dealing  with  syntax  but  are  only  beginning 
to  understand  meaning.  We  must  learn  how 
to  deal  with  the  semantics  of 
communications  to  gain  full  meaning.  And 
until  we  reach  that  level  of  understanding 
we  will  continue  to  have  several  false  starts. 
There  are  limitations  in  what  we  can  achieve 
with  current  software  technology.  But  even 
if  there  were  not,  you  can't  correct  a  problem 
until  you  know  what  it  is. 

CAPT  James  Hollenbach  USN  (DMSO) 
—  CAPT  Hollenbach  explained  that 
DMSO's  role  in  the  M&S  community  is  not 
to  build  simulations  but  to  serve  the  broader 
community  as  a  catalyst  in  accordance  with 
the  city  planning  analogy  to  the  high  level 
architecture.  The  High  Level  Architecture 
(HLA)  was  adopted  to  provide  a  more 
complete  technical  framework  for 
interoperability  and  reuse  which  overcomes 
the  limitations  of  DIS  and  Aggregate  Level 
Simulation  Protocol  (ALSP).  DIS  and 
ALSP  are  point  solutions  and  are 
inappropriate  for  many  applications, 
especially  when  you  need  live  systems 
interacting  with  virtual  and  constructive 
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simulations  for  analytical  or  training 
purposes.  The  HLA  offers  a  fuller  set  of 
capabilities  which  will  be  incorporated  in 
DIS  3.0  with  a  new  vision  document.  No 
single  simulation  will  be  able  to  satisfy  all 
requirements  of  the  training,  acquisition  or 
analytical  communities.  Instead,  the  DoD 
will  use  the  fundamental  approach  of 
composing  a  federation  of  simulations  and 
systems,  from  each  of  these  communities,  to 
answer  the  new  and  broader  questions  posed 
by  Dr.  Jones  and  others  from  the  joint  world. 
DMSO  definitely  wanted  the  analytical 
community  involved  and  requested 
aggressive  feedback  from  the  Workshop  on 
what  capabilities  are  required  in  the  HLA  to 
support  this  objective  from  the  analyst’s 
perspective.  He  then  summarized  a  list  of 
initiatives  that  DMSO  has  started  and 
challenged  the  analysts  represented  to  get 
engaged  in  these  activities.  These  initiatives 
include  seeking  common  data  standards  and 
data  exchange  formats,  developing  data 
quality  tools  to  include  security, 
understanding  behavioral  representations  in 
decision  making  and  understanding  the 
state-of-practice  in  computer  generated 
forces. 

Dr.  Ben  Wise  (SAIC)  —  Dr.  Wise  offered  a 
series  of  observations  from  a  hands-on- 
technologist  looking  up  at  ADS.  He  sees 
many  technical  problems;  some  with  easy 
solutions,  others  with  very  hard  solutions. 
He  suggested  that  cross  fertilization  for 
analysts  with  the  Testing  and  Evaluation 
(T&E)  community  would  be  a  good  bet  for 
both  communities.  And  as  for  federations 
composed  of  validated  models  interacting 
with  other  validated  models,  he  observed 
that  there  is  little  assurance  that  the  results 
obtained  from  the  federation  of  models  will 
be  valid.  Before  reasonable  conclusions  can 
be  reached,  more  rigorous  thinking  is 
needed  by  the  analytical  community  to 
capture  meta  data  on  overarching  validation 


within  a  federation  of  models.  Particular 
attention  must  be  paid  to  how  models  can  be 
"called"  by  other  models,  taking  note  of  time 
sequencing  issues.  A  reliable  time 
management  scheme  is  needed  and  analysts 
must  carefully  design  scenarios  to  reduce 
data  latency  across  networks.  Dr.  Wise  also 
observed  that  repeatability  will  be  a  real 
technical  challenge.  Real  people  don't  repeat 
each  other’s  behavior.  But  this  issue  also  has 
its  upside.  Look  at  any  federation  with 
people  in  the  loop  as  an  opportunity  to  study 
how  people  change  behavior  or  outcomes 
under  differing  scenarios.  Soft  factors,  such 
as  these,  are  little  observed  in  this  class  of 
simulations  and  will  be  very  useful  data  for 
representing  human  factors  in  other 
simulations  for  analytical  purposes.  Data 
produced  in  this  way  may  be  very  similar  to 
data  obtained  from  WW  n  and  Vietnam,  and 
people  were  able  to  do  good  analysis.  He 
concluded  with  a  challenge  to  the  working 
groups  to  understand  why  we  put  people  in 
the  loop  and  how  would  we  do  analysis  with 
cogent  software  in  the  loop  rather  than 
attribute-based  models. 

Dr.  Paul  Davis  (RAND)  —  Dr.  Davis’ 
comments  were  motivated  by  Dr.  Jones’ 
metaphor  of  city  planning.  He  speculated 
what  SimCity  would  be  like  if  it  had  been 
developed  with  current  DoD  attitudes.  He 
imagined  buying  it,  installing  it  on  his 
Macintosh,  opening  the  application  and 
discovering  to  his  horror  that  he  couldn't 
build  a  city  without  first  spending  six 
months  to  build  a  high  resolution  data  base 
of  bricks  and  mortar.  Why?  Because  in  his 
nightmare,  the  developer  had  a  vision  that 
focused  exclusively  on  building  from  the 
bottom  up  (starting  with  the  molecular?).  He 
found  this  very  troubling  because  in  the  real 
world,  almost  all  really  important  decisions 
need  to  be  formed  by  aggregate  models, 
which  can  actually  be  understood.  His  point: 
"Where  in  the  DoD  program  does  one  see 
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any  appreciation  of  aggregate  models  and 
the  need  for  them?"  He  also  observed  that 
there  must  be  a  new  law  of  large  numbers.  It 
is  obvious  from  the  way  people  talk  that 
there  must  be  a  law  that  says  as  the  number 
of  bad  models  and  databases  connected 
increases,  the  quality  of  the  overall 
simulation  improves,  converging  to  truth." 
His  point:  "Why  would  a  good  analyst  want 
to  put  his  seal  of  approval  on  conclusions 
based  on  mnning  some  ungodly  kluge  of 
models  that  he  neither  comprehends  nor 
tmsts?"  After  making  these  pointed 
comments,  Dr.  Davis  indicated  that  he  was 
actually  very  bullish  on  ADS.  He 
commented  that  ADS  technology  is  creating 
wonderful  opportunities  for  analysts  to  work 
closely  with  real  world  operators  and 
problems;  to  conduct  experiments  yielding 
insights  about  very  complex  phenomena;  to 
help  calibrate  models  using  the  latest  data; 
and  to  test  various  contingency  plans.  Key  to 
exploiting  these  opportunities  is  to  design 
ADS  experiments  to  serve  the  mechanics  of 
analysis  such  as  the  exploration  of  new 
tactics  or  model  calibration,  and,  to  use 
constructive  models  for  the  real  analyses 
such  as  when  extensive  uncertainty  analysis 
is  required  and  to  reach  robust  conclusions. 
In  his  view,  ADS  should  not  be  used  directly 
to  do  analysis.  Indeed,  he  believes  that  the 
analyst  community  should  strongly  fight 
pressure  to  impose  that  mode  upon  them; 
remembering  that  using  ADS  directly  means 
having  to  say:  "Well.  I  don't  understand 
what  went  on  in  the  simulation  or  whether  it 
is  correct,  but  what  the  "model"  said  is  that 
"x"  is  better  than  "y."  This  violates  one  of 
the  most  important  commandments  of 
analysis. 

Mr.  Mike  Bauman  (TRAC)  —  Mr. 
Bauman's  comments  were  on  the  role  of  the 
analyst  within  ADS.  He  indicated  that 
relevant  and  credible  analysis  is  required  to 
support  decisions  within  DoD.  And  while 


dozens  of  simulations  are  putting  out  data, 
the  most  important  aspect  of  any  simulation 
is  the  analyst.  Running  a  simulation  and 
looking  at  the  output  does  not  equal 
analysis.  The  analyst  must  understand  the 
algorithms  within  the  simulation  to  include 
causality,  etc.  He  sees  the  biggest  challenge 
to  the  analytical  community  as  an  ethical 
one  —  not  a  technical  one.  While  many 
people  are  viewing  ADS  as  shrink  wrapped 
software,  it  cannot  be  used  in  credible 
analysis  without  understanding  what  the 
model  does.  And  that  will  be  a  very  difficult 
task  with  federations  composed  of 
simulations  from  multiple  Services  and  of 
varying  granularity.  A  greater  burden  is 
placed  on  the  analyst  to  understand  how  we 
collect,  interpret  and  share  data.  Analysts 
must  make  sure  that  they  have  a  good  match 
between  problems  and  available  simulations 
and  to  be  responsible  to  stand  up  ethically  if 
this  isn't  the  case.  This  means  that  they  must 
have  the  courage  to  tell  leaders  that  the 
simulation  doesn't  support  answering  the 
questions  they  have  posed.  The  community 
has  a  real  challenge  in  maintaining 
confidence  of  leadership  in  the  new  way  we 
are  using  simulations.  Analysts  are  called  in 
on  an  issue  when  there  are  unknown  risks. 
They  are  asked  to  help  understand  the 
uncertainty.  So  they  must  be  able  to  reach 
into  models  or  simulations  and  manage 
change.  This  is  a  major  problem  with  ADS. 
The  conmiunity  must  be  particularly  vigilant 
in  how  we  present  "visualization"  to  the 
users  of  simulations.  A  realistic  picture  must 
be  presented  to  operators  and  actors. 

Dr.  Hank  Dubin  (OPTEC)  —  Dr.  Dubin 
commented  on  ADS  issues  relevant  to  the 
T&E  domain.  He  has  heard  a  lot  of 
discussion  about  immediate  significant  cost 
savings  in  the  training  community,  but  very 
little  has  yet  been  proven  with  ADS. 
On-the-other-hand,  the  T&E  community  has 
used  interactive  simulations  for  over  30 
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years  in  live-to-Iive  simulations,  live-to- 
constructive  and  live-to-virtual.  And  the 
T&E  community  is  moving  forward  with 
many  new  concepts  to  enhance  realism  and 
affordability  of  T&E.  He  cautioned  that  the 
ADS  approach  may  be  the  most  expensive 
way  to  do  simulations;  with  return  on 
investment  often  dependent  upon  reuse.  He 
asked  that  ADS  capabilities  not  be  bought 
just  because  they  are  fun  to  play  with.  He 
charged  the  working  groups  to  ask  the 
following  three  questions: 

(1)  Is  ADS  the  best  way  to  solve  the 
customer's  problem? 

(2)  What  are  the  trade-offs  among 
cost,  schedule  and  performance  in 
building  the  needed  pieces  and 
integration  for  the  ADS  federation? 

(3)  How  much  confidence  do  we 
have  in  the  selected  ADS  federation? 
He  offered  Dubin’s  ADS  self 
fulfilling  prophecy  theory. 

"The  more  complex  the  federation,  the  more 
interactive  in  aggregate,  the  more  credible 
the  results."  Since  we  don't  know  or 
understand  what  is  going  on,  we  have  no 
basis  for  discrediting  the  outcomes.  He  went 
on  to  comment  that  testing  has  long  been  a 
form  of  interactive  simulation  — ^test  items 
interacting  with  test  conditions  (some  of 
which  are  simulated).  And  since  testing  is 
already  a  development  tool,  better  and  more 
integration  with  other  simulations  has  the 
potential  to  become  a  much  better 
development  tool,  and,  as  such,  will  offer 
the  potential  for  earlier  checkout  of  the 
human-machine  interface,  system  design, 
and  procedural  considerations.  While  other 
simulations  cannot  totally  replace  testing,  if 
properly  integrated,  they  can  make  testing 
more  realistic  while  still  affordable.  Most 
agree  that  we  can't  use  simulation  alone  to 


prove  out  capabilities  of  a  system  as 
fabricated  in  hands  of  typical  users  and  in  a 
complex  environment,  such  as  combat.  One 
can  use  simulations  to  conclude  that  IF  the 
system  meets  all  design  parameters  and  IF 
operated  in  accordance  with  prescribed 
procedures,  then  it  SHOULD  perform  as 
indicated  by  the  simulation.  So  if  simulation 
cannot  totally  replace  testing,  what  is  its 
potential  value  to  T&E?  ADS  can  create 
effects  on  systems  under  test  which 
otherwise  would  not  be  practical.  ADS  can 
help  check  out  designs,  procedures  and 
components  before  the  system  is  fabricated 
to  do  much  better  at  reducing  risk.  But  this 
potential  value  in  reducing  risk  affordably  is 
not  yet  proven. 

CAPT  Drew  Beasley  USN  (JSIMS)  — 
CAPT  Beasley  commented  on  the  joint 
training  simulation  community.  JSIMS  is  a 
joint  training  simulation  under  development 
that  is  expected  to  provide  a  training  vehicle 
for  a  wide,  diverse  group  of  users.  The 
customers  for  JSIMS  are  the  CINCs,  the 
JWFC  and  the  Services  with  training 
requirements  set  by  the  JWFC.  As 
development  advances,  there  are  several 
areas  that  need  to  be  better  understood  and 
help  is  requested  from  the  analjrtical 
conununity.  One  of  these  is  analysis  support 
for  exercise  planning.  Currently,  analysis 
support  is  done  manually  based  upon  the 
lessons  remembered  from  the  last  exercise 
or  last  war.  Surely  there  are  better  ways  to 
accomplish  this.  Another  is  analysis  support 
for  After  Action  Reviews  (AARs).  Are  there 
better  ways  to  indicate  what  data  needs  to  be 
collected  and  once  it  is  collected  how  do  you 
cull  useful  "stuff  from  the  reams  of  data? 
We  need  to  examine  ways  to  adjust  analysis 
to  support  training  objectives.  ■\^at  are  the 
relevant  measures  of  merit  (MoMs)?  These 
questions  imply  that  we  need  to  link  the 
experimental  design  with  the  data  collection 
effort  and  the  analysis  effort  —  something 
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in  which  the  analysis  community  should  be 
well  versed.  The  JSIMS  AAR  capability 
needs  to  be  able  to  identify  cause  and  effect 
via  analytical  tools  to  predict  relevant  issues 
that  need  to  be  considered  for  mission 
planning  and  mission  rehearsal.  For 
example,  is  doctrine  sound  or  would  a 
change  help?  Or  do  we  have  the  right 
weapon  mix?  These  are  important  questions 
the  JSIMS  project  office  is  addressing  and  it 
is  important  that  this  Workshop  identify 
common  objectives  for  the  training  and 
analytical  communities  that  make  sense  to 
share  and  work  towards. 

5.  Working  Group  Synthesis.  ADSA  '96 
Proceedings  were  developed  that  contain 
detailed  reports  from  each  WG.  But  for  the 
purposes  of  this  section,  we  will  draw  upon 
the  Synthesis  Panel  report  developed  by  Dr. 
Stu  Starr  and  his  panel  members.  These 
panel  members  participated  as  active 
members  of  one  of  the  other  six  working 
groups  of  the  Workshop  as  well  as  the 
Synthesis  Panel. 

The  nature  of  the  problem.  One  of  the  first 
questions  that  emerged  during  the 
deliberations  was  the  issue  of  the  definition 
of  the  term  ADS.  The  importance  of 
developing  a  definition  that  was  widely 
accepted  by  the  community  was  suggested 
by  one  of  the  papers  in  the  read-ahead 
package,  "What  in  the  World  is  ADS?"  by 
Sikora  and  Goose.  By  invoking  the  image  of 
the  Tower  of  Babel  in  their  paper,  the 
authors  suggested  the  confusion  that  can 
ensue  if  the  participants  in  the  dialogue  lack 
a  common  frame  of  reference  and 
vocabulary. 

In  recognition  of  this  issue,  the  Synthesis 
Panel  spent  a  moderate  amount  of  time 
exploring  the  question,  as  did  many  of  the 
other  panels.  It  concluded  that  the  strawman 
definition  in  DoD  5C)00.59-M  is  inadequate. 


A  taxonomic  approach  to  the  question  shows 
some  promise  and  is  discussed  in  the 
findings  and  recommendations  below. 

Several  of  the  working  groups  spent 
considerable  time  identifying  and  discussing 
why  the  analysis  community  has  been  slow 
to  embrace  ADS.  These  issues  can  be 
loosely  aggregated  into  two  categories: 
quality  and  use. 

In  the  area  of  Quality,  the  working  groups 
cited  the  following  issues: 

•  Ambiguity.  Since  many  of  the 
members  of  the  analysis 
community  are  not  sure  what  ADS 
is,  they  are  not  sure  how  useful  it  is 
to  them. 

•  Replicability.  Since  many 

examples  of  ADS  feature 
humans-in-the-loop,  there  is 
concern  among  some  analysts 
about  the  ability  to  replicate 
results.  In  particular,  there  is 
concern  about  coping  with  the 
effects  of  subject  learning  and  the 
impact  of  differences  in  subject 
background,  experience, 

proficiency,  and  morale. 

•  Credibility.  It  was  acknowledged 
by  several  working  groups  that 
Verification,  Validation  and 
Accreditation  (VV&A)  of  ADS 
pose  particular  challenges. 

•  Suitability  for  Some  Issues.  Several 
working  groups  noted  that  there  are 
classes  of  analysis  problems  for 
which  ADS  may  not  be  suitable. 
This  can  stem  from  the  limited 
knowledge  base  that  may  exist  for 
a  class  of  problems  (e.g.,  studies  of 
new  classes  of  ships  that  are  still  in 
the  preliminary,  conceptual  stage) 
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or  the  limited  precision  that  one 
can  hope  to  achieve  using  some 
classes  of  ADS  (due  to  the 
likelihood  of  relatively  small 
sample  sizes). 

In  the  area  of  Use  the  working  groups  cited 

the  following  issues: 

•  Cost/Resource  Implications. 

Several  working  groups  observed 
that  it  is  likely  to  be  relatively 
expensive  to  acquire,  update  and 
employ  ADS  in  comparison  to 
more  traditional  analysis  tools.  In 
addition,  from  a  resource 
perspective,  it  is  often  difficult  to 
obtain  and  train  suitable  test 
subjects. 

•  Time/Schedule  Implications. 

Numerous  Working  Groups 
expressed  concern  about  the 
extensive  time  implications  of 
ADS.  This  includes  the  time  to 
create  an  ADS  capability,  setup 
conditions  for  a  specific  analysis, 
perform  test  runs  and  reduce  the 
data  that  is  generated  by  exercising 
an  ADS.  These  time  considerations 
can  preclude  use  of  ADS  when  the 
analyst  is  given  little  lead  or 
execution  time  (e.g..  Program 
Objective  Memorandum  issue 
analyses). 

•  Data  Implications.  Several 
participants  identified  a  broad 
spectrum  of  data  issues  associated 
with  ADS.  These  include  problems 
associated  with  acquiring  data 
needed  to  setup  and  execute  ADS; 
deciding  what  data  to  collect  and 
collecting  it  (note:  it  may  be 
difficult  to  introduce  ad  hoc 
"hooks"  into  an  ADS  to  collect 


desired  data);  and  storing  and 
processing  the  resulting  data  (e.g., 
some  of  the  data  of  interest  may  be 
spoken  by  the  test  subjects). 

•  Exercising  the  ADS,  Technically. 
Several  ADS  practitioners 
identified  the  problems  that  they 
encountered  in  technically 
exercising  an  ADS.  To  illustrate 
the  problem  of  control,  a 
representative  from  the  Theater  Air 
C2  Simulation  Facility  (TACCSF) 
recalled  an  incident  where  a 
frustrated  AW  ACS  subject 
unrealistically  walked  from  his 
simulated  node  to  a  simulated 
ground  node  to  resolve  a  concept 
of  operations  issue.  Second,  the 
distribution  of  nodes  can  pose  time 
synchronization  issues  (e.g.,  the 
use  of  satellite  communications 
can  give  rise  to  latencies  that  are 
unacceptable  in  selected  classes  of 
analyses  such  as  real  time  and 
extended  air  defense  problems). 
Third,  several  practitioners  pointed 
out  that  interconnection  of 
heterogeneous  nodes  is  still  in  its 
infancy.  The  Distributed 
Interactive  Simulation  (DIS) 
standards/protocols  are  still 
incomplete  and  evolving.  In 
addition,  the  ambiguity  in  existing 
standards/protocols  is  such  that 
two  nodes  can  claim  to  be  DIS 
compatible  and  still  not  be  able  to 
interoperate.  Finally,  the  absence 
of  multi-level  security  solutions 
makes  it  difficult  to  conduct 
experiments  with  ADS  featuring 
nodes  at  different  classification 
levels. 

•  Administrative  Burdens.  ADS  can 
pose  administrative  burdens  that 
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are  far  greater  than  those  normally 
encountered  by  an  analyst.  For 
example,  if  the  analyst  is  truly  to 
leverage  the  resources  of  others 
(either  those  of  other  analysts  or  of 
other  communities  such  as  training 
or  acquisition),  it  will  entail 
extensive  cross-community/ 

organization  coordination.  As  an 
ancillary  problem,  it  is  generally 
onerous  to  schedule  the  facilities  of 
others,  particularly  when  the 
facilities  are  widely  distributed.  In 
addition,  for  those  ADS  that 
feature  HUL,  it  can  be  quite 
difficult  arranging  for  test  subjects 
(i.e.,  getting  people  with  the  right 
experience,  knowledge  and  skills 
for  the  times  needed).  Lastly, 
configuration  management  poses 
problems  since  it  must  be 
maintained  as  facilities  are 
modified  to  meet  analysis  needs 
and  to  track  real  world 
evolution  of  systems. 

•  Ejficient,  Effective  Exercising  of 
ADS.  In  order  to  employ  ADS 
efficiently  and  effectively,  it  is 
necessary  to  devise  and  implement 
experimental  designs  that  provide 
confidence  levels  consistent  with 
the  issues  under  study  and  to 
contend  with  potential  confounding 
factors  that  may  be  unique  to  ADS 
(e.g.,  compensate  for  potential 
learning  by  test  subjects). 

In  view  of  the  many  issues  associated  with 
ADS,  the  most  natural  question  is:  "Why 
should  analysts  bother  with  ADS?"  The 
Workshop  identified  several  significant 
opportunities  implicit  within  ADS  that 
suggest  that  it  is  well  worth  the  time  of  the 
analysis  community  to  explore  the 


application  of  ADS  to  a  broad  range  of 
analyses. 

First,  several  working  groups  reported  that 
ADS  has  the  potential  to  substantially 
enhance  analytic  support  to  a  number  of 
application  areas.  For  example,  in  the  area 
of  operations,  ADS  has  the  potential  to 
enhance  dramatically  the  quality  of  "what  if 
analyses.  As  an  illustration,  an  extension  of 
ARPA's  Project  Odin  technology  might 
enable  operational  commanders  to  visualize 
the  latest  intelligence  data  and  experiment 
with  alternative  tactics  and  procedures.  In 
the  area  of  acquisition,  it  was  noted  that  the 
state-of-the-practice  has  enabled  several 
commercial  firms  to  improve  the  quality  of 
their  products  (e.g.,  Boeing's  use  of 
(CATIA)  to  develop  the  "snap  together" 
777).  ARPA  is  at  the  state-of-the-art  level  in 
this  application  and  they  are  optimistic  that 
it  can  be  used  to  reduce  the  cost  and 
schedule  of  an  acquisition  while  enhancing 
the  quality  of  the  product  (e.g.,  the 
simulation  based  design  activities  for 
acquiring  future  naval  platforms). 

Second,  ADS  may  enable  analysts  to 
address  issues  in  a  richer  context  than  they 
otherwise  could.  As  illustrations,  note  that 
the  use  of  HUL  makes  ADS  an  attractive 
candidate  to  treat  human  interactions 
explicitly  and  to  consider  information 
aspects  of  warfare.  In  addition,  the  ability  to 
network  the  tools  from  several  Services 
suggests  that  it  should  help  to  address  joint 
issues  more  credibly. 

Third,  technology  trends  are  providing  new 
opportunities.  The  Information  Technology 
(U)  that  underlies  ADS  is  being  enhanced 
dramatically  by  the  commercial  sector  (e.g., 
substantial  improvements  in  processing 
speeds,  storage  size,  and  visualization  tools 
at  significantly  reduced  costs).  In  addition, 
efforts  are  underway  in  the  defense 
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community,  such  as  the  HLA,  to  promote 
interoperability  and  reuse.  The  net  effect  of 
these  trends  is  to  suggest  that  ADS  will 
become  increasingly  more  capable,  cheaper 
to  create  and  easier  to  use  and  reuse. 

Fourth,  as  observed  by  Dr.  Jones,  the 
analytic  community  has  the  opportunity  to 
take  advantage  of  the  extensive  ADS 
investments  of  other  communities  (most 
notably  the  training  community).  K  done 
properly,  this  could  provide  the  analysis 
community  with  ready  access  to  expertise 
(e.g.,  knowledge  on  how  to  use  ADS 
effectively  and  efficiently),  resources  (e.g., 
the  opportunity  to  use  existing  ADS)  and 
needed  data  and  parameters. 

Fifth,  Mike  Bauman,  TRAC-Ft. 
Leavenworth,  and  Gene  Visco,  FS,  ODUSA 
(OR),  observed  that  in  recent  years  we  have 
seen  the  rise  of  "computer  driven  analysis." 
They  postulated  that  ADS  poses  the 
opportunity  for  analysts  to  move  to 
"computer  assisted  analysis"  and  thereby 
"return  to  our  roots." 

Sixth,  it  was  observed  by  several 
participants,  that  decision  makers  are  being 
exposed  to  ADS  through  its  application  by 
other  communities.  Thus,  it  would  not  be 
surprising  if  these  decision  makers  were  to 
mandate  the  selective  use  of  ADS  by 
analysts  to  capture  many  of  its  attractive 
characteristics  (e.g.,  explicit  treatment  of 
human  behavior;  interactive  features;  high 
quality  visual  features). 

Finally,  Jack  Thorpe,  SAIC,  made  two 
observations  about  ADS.  He  predicted  that 
“In  ten  years,  everything  can  be  connected 
to  everything  else  and,  given  the  spirit  of 
experimentation  that  exists  within  the 
analytic  community,  it  will!"  Thus,  he 
opined,  ADS  is  coming  and  it  will  be 
pervasive  in  the  defense  conununity.  In 


addition,  he  speculated  that  ADS  may  be  the 
precursor  to  full  "instrumentation"  of  every 
combatant  on  the  battlefield.  If  that  comes  to 
pass,  it  will  lead  to  unprecedented  access  to 
timely  data  for  understanding  warfare.  By 
analyzing  these  data,  analysts  will  have  an 
unprecedented  opportunity  to  formulate  and 
validate  models  of  conflict. 

Findings  and  recommendations.  Based  on 
the  insights  that  the  Synthesis  Panel  gained 
from  participation  on  working  groups  and 
Synthesis  Panel  discussions,  a  total  of  four 
areas  were  identified  that  the  panel  members 
felt  were  of  overarching  importance.  For 
each  of  these  areas,  the  Synthesis  Panel 
developed  findings  and  a  set  of 
recommendations.  In  each  case,  the 
Synthesis  Panel  identified  the  organizations 
that  they  believe  should  take  the  lead  in 
implementing  the  recommendations.  All 
these  areas  are  subject  to  modification  and 
addition  as  the  Proceedings  are  completed 
with  final  WG  reports. 

ADS  Definition.  The  Sjmthesis  Panel 
observed  that  there  is  no  existent  definition 
of  ADS  that  is  deemed  adequate  by  the 
Workshop  participants.  The  Panel  found  that 
this  lack  of  an  adequate  definition  limits  the 
analytic  community’s  ability  to  understand 
the  nature  of  the  capability  (i.e.,  to 
appreciate  its  current  or  Mure  attributes  or 
limitations).  This,  in  turn,  limits  the  analytic 
community's  ability  to  plan  for  the  effective 
application  of  both  ADS  and  non-ADS 
M&S  capabilities. 

Consequently,  the  Synthesis  Panel 
recommends  that  a  taxonomy  should  be 
developed  that  defines  and  clarifies  the 
nature  of  ADS  and  illuminates  its 
relationship  to  related  concepts  such  as  DIS. 
Ed  Brady’s,  FS,  proffered  definition  can 
serve  as  Ae  starting  point.  The  MORS  ADS 
Senior  Advisory  Group  (SAG)  would  be  an 
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appropriate  organization  to  draft  this 
taxonomy  and  then  pass  it  on  to  the  Defense 
Modeling  and  Simulation  Office  (DMSO) 
for  approval  and  dissemination  throughout 
the  coimnunity. 

Community  Leadership.  The  Synthesis 
Panel  found  that  the  challenges  associated 
with  ADS  transcend  the  abilities  and 
resources  of  the  individual  analyst.  If 
analysts  are  to  use  ADS  effectively,  a 
paradigm  shift  must  occur  to  enhance 
collaborative  analysis.  This  concept,  which 
was  put  forth  by  the  Analysis  and 
Requirements  WG,  rests  on  pillars  of  cross¬ 
community  shared  data,  tools  (e.g.,  scenario 
generation  tools,  visualization  tools), 
security,  MoMs  and  an  integrated  family  of 
analysis  techniques.  The  last  of  these  pillars 
should  be  emphasized.  Most  of  the  working 
groups  observed  that  ADS  should  not  be 
viewed  as  a  "stand-alone"  capability  that  can 
be  applied  to  all  analytic  problems.  In 
general,  it  must  be  harmonized  and 
orchestrated  with  other  analysis  techniques 
to  compensate  for  selected  ADS  features 
(e.g.,  extensive  time  to  create  and  execute) 
and  employed  where  appropriate. 

In  order  to  bring  about  this  paradigm  shift, 
key  analytic  organizations  and  the  analytic 
community  must  provide  needed  stimulation 
and  leadership.  A  partial  listing  of 
recommended  actions  to  be  undertaken  by 
the  Services,  the  Joint  Staff  and  the  OSD 
include  the  following: 

•  Providing  Incentives  to  Use  ADS. 
This  would  include  providing 
appropriate  resources  to  analysts 
and  promoting  those  analysts  who 
employ  ADS  appropriately  and 
effectively. 

•  Making  Necessary  Investments  in 
Community  Infrastructure  and 


Accreditation  Efforts.  It  was 
observed  that  the  analysis 
community  is  normally  limited  to 
"chump  change."  To  provide  the 
resources  needed  to  develop  and 
VV&A  ADS,  several  members  of 
the  Synthesis  Panel  recommended 
that  a  "PM  Analytic  ADS" 
organization  be  established  and 
supported  with  adequate  budget 
line  items.  This  recommendation 
should  be  assessed  further  to 
establish  its  feasibility  and 
desirability. 

•  Facilitating  Needed  Education, 
Training.  The  cornerstone  to 
effective  creation  and  use  of  ADS 
is  the  education  and  training  of  the 
analyst  and  the  decision  maker.  In 
view  of  the  criticality  of  this  step, 
it  is  amplified  as  a  separate  issue 
below. 

•  Forming  Teams  With  the 

Necessary  Mix  of  Skills.  The 
effective  creation  and  use  of  ADS 
demands  a  team  effort. 
Representative  skills  include 
operations  research  (to  help  frame 
and  scope  the  problem), 
experimental  design  (to  formulate 
the  experimental  design  matrices 
and  support  the  analysis  process), 
program  management  (to  perform 
the  myriad  management  and 
administrative  tasks)  and  data 
mining  (to  acquire,  process  and 

store  data  associated  with  ADS). 

ADS  Plan  for  the  Analysis  Community. 

Once  the  analysis  community  becomes 

committed  to  the  enhanced  collaborative 

analysis  paradigm,  it  will  need  a  plan  to 
guide  its  actions.  Historically,  the  analysis 
community  has  been  reticent  to  take  this 
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step.  For  example,  several  years  ago,  DMSO 
sponsored  a  series  of  workshops  to  stimulate 
the  various  communities  to  identify  their 
needs  for  advanced  M«&S.  While  several 
communities  embraced  this  opportunity 
(e.g.,  training,  T&E),  the  analysis 
community  demurred.  The  Synthesis  Panel 
further  observed  that  there  is  no  long  range 
vision  to  guide  the  allocation  of  limited 
M&S  development  resources  to  support  the 
analyst.  Consequently,  the  analysis 
community  tends  to  be  excessively  reactive 
to  near-term  needs  at  the  expense  of  needed 
longer-term  investments. 

The  Synthesis  Panel  found  that  the  absence 
of  an  "ADS  Plan  for  the  Analysis 
Community"  has  resulted  in  fragmented 
action  within  the  analysis  conununity  and 
limited  the  leveraging  of  the  resources  and 
expertise  that  are  resident  in  other 
communities  (e.g.,  training,  acquisition).  To 
ameliorate  those  shortfalls,  the  Analysis 
Council  that  has  been  formed  under  the 
Executive  Committee  for  M&S  (EXCIMS) 
must  take  decisive  action.  It  should  develop 
the  vision  for  the  next  generation  of  M&S 
tools  for  the  analyst  and  draft  an  associated 
"ADS  Plan  for  the  Analysis  Community."  In 
the  view  of  the  need  for  timely  action,  these 
products  should  be  drafted  and  circulated  for 
community  coordination  no  later  than  1 
February  1997.  To  ensure  that  the 
community  takes  full  advantage  of  the 
opportunities  implicit  in  commercial 
developments,  the  plan  should  explicitly 
identify  steps  to  ensure  maximum  use  and 
compatibility  between  Commercial-Off- 
The-Shelf  (COTS)  products  and  ADS. 

Community  Education  Training  and 
Experience.  The  Synthesis  Panel  found  that 
a  broad  set  of  skills,  knowledge  and 
expertise  will  be  needed  by  the  community 
if  it  is  to  be  able  to  appreciate  the 
capabilities  and  limitations  of  ADS  and  to 


employ  ADS  effectively  and  efficiently.  It 
must  be  emphasized  that  the  community  in 
question  subsumes  both  analysts  and 
decision  makers.  In  the  absence  of 
appropriate  education,  training  and 
experience,  the  results  that  analysts  derive 
from  the  application  of  ADS  are  likely  to  be 
suspect  and  the  decision  makers' 
understanding  of  the  meaning  and  validity  of 
those  results  is  likely  to  be  limited. 

To  deal  with  these  issues,  the  Synthesis 
Panel  put  forth  several  recommendations. 
First,  key  Service  schools  should  develop 
curricula  and  offer  a  sub-specialty  of  "ADS 
Analyst."  Candidate  schools  include  the 
Naval  Postgraduate  School  (NPS),  the  Air 
Force  Institute  of  Technology  (AFIT)  and 
the  Army  Logistics  Management  College 
(ALMC).  Second,  assignments  to  "ADS 
Analyst"  positions  should  be  of  suitable 
duration  and  consistency  to  ensure  that 
necessary  experience  is  acquired  and 
sustained.  Thus,  for  example,  once  an 
analyst  has  qualified  as  an  "ADS  Analyst"  at 
one  of  the  above  institutions,  his/her  next 
assignment  should  be  to  an  ADS  activity  or 
to  an  organization  which  may  perform 
collaborative  analysis  employing  ADS.  Note 
that  some  Services  currently  assign  recent 
graduates  to  field  assignments. 
Subsequently,  a  sequence  of  assignments 
should  be  envisioned  that  help  the  ADS 
Analyst  to  continue  to  hone  and  apply 
his/her  skills. 

In  support  of  decision  maker  education  and 
training,  appropriate  programs  on  ADS 
should  be  developed  and  offered  at 
institutions  such  as  the  Defense  Systems 
Management  College  (DSMC),  the  Defense 
Acquisition  University  (DAU),  the  National 
Defense  University  (NDU)  and  the  Senior 
Service  Defense  Schools.  Thought  should  be 
given  to  updating  DSMC’s  Handbook  on 
M&S  for  the  acquisition  community  to 
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highlight  the  challenges  and  opportunities 
associated  with  ADS. 

Qualitative  assessment  of  the  Workshop. 

The  TOR  identified  four  objectives  for  the 
Workshop.  To  establish  a  qualitative 
measure  of  the  effectiveness  of  the 
Workshop,  the  Synthesis  Panel  informally 
assessed  the  worldng  groups  against  these 
four  objectives,  based  on  its  observations  of 
their  operations  and  products.  These 
assessments  were  in  the  form  of  a  color  (i.e., 
"green"  implied  fairly  complete  achievement 
of  the  objective,  "amber"  implied  moderate 
achievement  of  the  objective  and  "red" 
implied  a  failure  to  achieve  the  objective). 

The  first  objective  was  to  assess  the  utility 
of  current  and  future  (3-5  years  out)  ADS- 
based  capabilities  for  analytic  applications, 
especially  possible  areas  of  new  analytic 
capabilities.  The  S5mthesis  Panel  rated  this 
as  "amber."  There  was  extensive 
consideration  of  the  current  time  frame  but 
less  about  the  future  because  of  the  limited 
understanding  about  the  likely  nature  of 
ADS  by  the  turn  of  the  century. 

The  second  objective  was  to  define  areas  of 
needed  enhancements  of  ADS-based 


capabilities  to  increase  their  utility  to  the 
military  analytic  community.  The  Synthesis 
Panel  rated  this  as  "amber."  It  was  felt  that 
the  absence  of  an  agreed  definition  for  ADS 
limited  this  discussion,  although  a  number 
of  panels  put  forth  excellent  suggestions  for 
increasing  the  utility  of  ADS  to  the  military 
analytic  community. 

The  third  objective  was  to  increase  the 
military  analytic  community's  understanding 
of  current  ADS-based  capabilities  and  near 
term  improvement  programs.  The  Synthesis 
Panel  rated  this  as  "green."  This  was  due  to 
the  inclusion  of  descriptive  material  in  the 
read-ahead  package,  the  material  covered  in 
selected  plenary  presentations,  the 
scheduling  of  ADS  presentations  within 
selected  panels,  and  the  ad  hoc  scheduling  of 
a  presentation  on  the  HLA  to  the  Workshop. 

The  fourth  objective  was  to  increase 
interactions  between  the  military  analytic 
community  and  developers  of  ADS-based 
capabilities.  The  Synthesis  Panel  rated  this 
as  "green."  This  was  due  to  the  mix  of 
participants  that  were  invited  to  the 
Workshop  and  the  opportunities  that  were 
provided  to  enable  them  to  have  a  fruitful 
dialogue. 
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MORS  Workshop 
Advanced  Distributed  Simulation 
(ADS)  for  Analysis 

Working  Group  1 
Battlefield  Effectiveness 
(Did  not  submit  a  report) 


MORS  Workshop 
Advanced  Distributed  Simulation 
(ADS)  for  Analysis 

Working  Group  2 
Material,  Systems  and  Acquisition 


Working  Group  Chair:  Lt  Col  Bob  Sheldon,  AFSAA 

Co-Chairs:  Dennis  Shea,  CNA 

Maj  Willie  McFadden,  TRADOC 
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MORS  ADS  Workshop  -  WG  2  Challenges 

•  ADS  is  not  a  new  methodology  for  analysis  —  all 
the  components,  including  human-in-the-loop 
(HITL),  have  been  available  to  analysts  for  years. 

•  Rather,  it  is  the  glue  that  allows  analysts  to  tie 
together  formerly  stand-alone  analytic  tools  into  a 
more  powerful  mechanism  for  modeling  complex 
problems. 


Although  the  components  of  the  technology  are  not  new,  the  challenge  is  to  integrate 
these  technologies  in  new  ways  and  to  ensure  that  the  resulting  applications  are 
appropriate  and  produce  results  that  are  meaningful  and  understandable. 

The  problem  is  made  more  difficult  because  the  models  and  simulations  that  are  being 
linked  together  were  developed  by  different  vendors,  over  many  years,  using  different 
technologies  and  were  not  designed  originally  to  interoperate  together  in  useful  and 
meaningful  ways. 
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To  ensure  that  ADS  becomes  useful  for 
analysis  in  acquisition  requires: 

•  Greater  attention  to  experimental  design  and  a 
disciplined  approach  to  data  collection  and 
analysis 

•  Affordable  V&V  techniques  for  linked 
heterogeneous  simulations 

•  Pre-planned  control  response  “playbook” 

-  To  handle  loss  of  link  or  simulations 
(continued) 


The  following  three  slides  summarize  the  essential  requirements  for  ADS  to  be  useful  for 
analysis  supporting  acquisition  decisions. 

To  ensure  that  ADS  is  useful  for  analysis,  requires  that  we  get  the  most  information  from  the 
few  ADS  experiments  that  we  can  conduct  practically  on  any  particular  issue.  Traditional 
design  of  experiment  techniques  can  help  us  stmcture  these  experiments  to  ensure  that  they 
are  conducted  in  the  most  efficient  manner. 

Establishing  the  credibility  of  ADS  through  V&V  will  be  critical  to  its  utility  and  ultimate 
acceptance  by  DoD.  It  is  relatively  easy  to  be  overcome  by  the  apparent  realism  of  some  of 
the  synthetic  environment  presentations  and  forget  that  the  models  underlying  these 
presentations  are  based  on  assumptions  and  calculations  that  may  or  may  not  be  appropriate 
for  the  application  at  hand.  As  the  model  outputs  become  more  and  more  complex,  analyzing 
the  credibility  of  the  model  outputs  becomes  more  challenging.  Even  if  the  models  and 
simulations  have  undergone  the  V&V  process  as  stand-alone  naodels,  linking  them  together  in 
an  ADS  experiment  may  lead  to  conflicts  and  errors  that  can  invalidate  the  model  output. 

When  using  ADS  in  practice,  the  exercise  control  team  needs  to  have  a  plan  in  place  to 
handle  the  inevitable  system  failures,  such  as  loss  of  the  netwoik,  other  datalinks,  or  if  a 
simulation  drops  off-line.  The  challenge  is  to  keep  system  failures  away  from  the  participants. 
Ideally,  exercise  control  can  keep  the  scenario  going  and  manage  to  smooth  over  minor 
software  and  hardware  problems  so  they  are  transparent  to  the  participants.  One  approach  that 
is  often  successful,  particularly  in  exercises  run  from  distributed  locations,  is  the  development 
of  a  controller  "playbook"  that  is  a  list  of  pre-planned  responses  that  are  designed  to  minimize 
the  effect  of  technical  problems. 
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To  ensure  that  ADS  becomes  useful  for 
analysis  in  acquisition  requires: 

•  Experiments  that  don’t  confound  acquisition 
objectives  with  training  objectives 

•  Improved  data  collection/reduction/analysis  tools 

•  Greater  attention  to  data  collection 

•  Training  of  both  analysts  and  military  decision 
makers  on  how  to  use  ADS  tools  and  results  from 
ADS  analyses 


The  high  cost  of  conducting  an  ADS  experiment/exercise  suggests  that  it  will  be  difficult  to 
justify  these  costs  for  analysis  purposes  alone.  Yet  using  an  ADS  experiment/exercise  to  both 
accomplish  training  and  to  analyze  the  effectiveness  of  a  new  system  can  pose  conflicting 
requirements.  The  objectives  of  these  two  uses  are  likely  to  be  different,  as  are  the  credibility 
requirements  (including  resolution  and  fidelity)  of  the  ADS  to  support  them. 

Training  applications  are  designed  to  promote  the  development  of  skills.  Analysis  applications, 
on  the  other  hand,  are  intended  to  lead  to  more  informed  decisions  about  equipment,  tactics, 
doctrine,  etc.  A  lot  of  training  objectives  can  be  accomplished  with  fairly  low  fidelity  models, 
even  models  that  are  not  completely  accturate.  The  military  services  have  been  using  the  CBS, 
AWSIM,  RESA  and  ENWGS  models  for  years  even  though  they  have  not  been  through  any 
formal  W&A.  The  minimiim  credibility  criteria  to  support  training  is  that  the  simulations 
provide  the  right  stimulus  for  the  training  audience  to  learn  better  skills  and  that  the  models  do 
not  induce  negative  training.  The  credibility  criteria  to  support  analysis  (e.g.,  to  support  an 
acquisition  decision)  are  generally  more  stringent.  Here  the  simulations  must  provide  realistic 
information  on  the  effectiveness  of  systems  in  a  potential  real  world  situation. 

To  make  effective  use  of  ADS  for  analysis  will  require  significant  improvements  in  automated 
tools  to  support  data  collection,  reduction  and  analysis.  Data  collection  cannot  be  viewed  as  an 
afterthought.  Without  careful  planning  to  set  the  filters  on  which  data  and  the  frequency  of  data 
that  will  be  collected,  the  ADS  exercise  will  quickly  overwhelm  the  data  recording  capabilities 
of  most  existing  systems.  Also,  these  current  systems  are  somewhat  awkward  and  painfully 
slow. 

Finally,  the  use  of  ADS  to  support  analysis  is  a  new  paradigm,  which  analysts  and  senior 
decision  makers  are  just  beginning  to  understand.  Significant  learning  by  both  parties  is 
required  to  understand  appropriate  applications,  to  stracture  the  ADS  experiments  and  to 
properly  interpret  the  results. 
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To  ensure  that  ADS  becomes  useful  for 
analysis  in  acquisition  requires: 

•  Better  documentation  of  the  analysis  process 

•  Incentives  to  try  ADS  in  specific  acquisition 
programs 

-  Services  conduct  AWE  experiments 

-  To  further  develop  ADS  technology 

•Use  ADS  as  part  of  a  larger  research  strategy  for 
analysis  in  acquisition 

•  Rethinking  DoD  acquisition  strategy 


Advanced  Warfighting  Experiments  (AWE)  is  an  Army  initiative  to  explore  innovative 
technologies  and  concepts. 

If  results  from  ADS  experiments  are  to  be  useful  to  decision  makers,  they  must  be 
understandable  and  be  able  to  withstand  scrutiny  by  outside  organizations.  This 
manHates  better  documentation  of  the  entire  analysis  process  than  we  have  today.  This 
should  include  the  following: 

•Clear  articulation  of  the  analysis  objectives 

•  The  rationale  for  using  ADS  including  selection  of  the  models  and  an 
assessment  of 

•  Their  credibility  and  appropriateness  for  the  particular  application 

•  Explanation  of  the  experimental  design  used 

•  Explanation  of  how  the  results  of  the  ADS  experiment  were  used  in  the 
analysis 

To  assess  the  credibility  of  the  models  and  simulations  requires  documentation  on  the 
assumptions  embedded  in  the  model,  the  variables  included  and  excluded  and  enough 
information  about  the  algorithms  and  overall  approach  to  convince  the  reader  that  the 
fidelity  and  modeled  processes  are  appropriate  for  the  application. 
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To  ensure  that  ADS  becomes  useful  for  analysis  in  acquisition  requires: 

notes  (Continued) 


Presently,  there  are  few  incentives  for  any  program  manager  to  try  ADS.  First,  ADS  is  a 
new  analytic  paradigm  and  few  understand  how  to  use  it  for  analysis.  It  isn  t  yet  clear 
that  it  will  be  helpful  for  analysis  or  whether  ADS  results  will  be  accepted  within  the 
formal  DoD  decision  process.  Therefore  no  program  manager  wants  to  spend  his/her 
funds  on  such  a  risky  venture.  Second,  because  DoD  lacks  an  M&S  infrastructure 
(models,  databases,  standards,  support  tools,  etc.)  to  support  users  and  developers  of  this 
technology,  there  wiD  be  significant  upfront  costs  to  try  ADS.  These  costs  are  too  large 
to  be  borne  by  any  individual  program.  The  funds  to  develop  the  inffastracture  to 
support  ADS  must  come  off  the  top  of  the  budget  or  result  from  a  tax  levied  on  each 
program.  To  learn  where  and  how  ADS  can  support  analysis,  each  of  the  services 
should  nominate  a  few  pilot  programs  to  experiment  with  ADS.  This  selection  process 
should  include  the  necessary  funding  to  support  the  ADS  experiments. 

Finally,  to  realize  the  benefits  of  ADS  may  require  rethinking  the  present  DoD 
acquisition  strategy  to  assess  whether  ADS  will  enable  some  of  the  traditional  steps  in 
the  acquisition  process  to  be  skipped  or  conducted  in  parallel.  We  need  some  analysis  to 
understand  where  to  seek  the  benefits  from  ADS  when  it  is  applied  to  acquisition 
programs.  The  analysis  should  examine  whether  ADS  can  reduce  costs,  help  field 
systems  faster  or  lead  to  more  informed  decisions  as  its  proponents  would  have  us 
believe.  The  results  from  a  few  pilot  programs  could  help  answer  these  questions. 


1.  When  should  ADS  be  used? 

•  Potential  to  be  applied  at  all  stages  of  the 
acquisition  cycle 

•  To  keep  the  warfighter  in  the  loop 

•  Visualization  to  improve  communications 

-  Between  waifighter,  designer,  analysts,  etc 

•  Better  define  requirements 

-  More  realistic/credible  in  terms  of  human  factors 


•  ADS  should  be  used  when  it  fits  the  problem  and  is  cost  effective. 

•  ADS  is  one  way  of  getting  the  war  fighter  in  the  loop  early  and  continuously. 

•  ADS  should  be  used  when  visualization  is  crucial  to  better  comprehension. 

•  ADS  can  be  used  to  accomplish  better  requirements  definition. 

•  There  are  several  types  of  applications  for  which  ADS  may  be  appropriate: 

1.  When  human  factors  are  a  critical  element  of  the  problem  and  representations  of  the 
human  interactions  through  a  constructive  model  are  insufficient.  For  example,  when  the 
analysts  are  trying  to  assess  how  warfighters  will  use  C4ISR  systems  in  their  tactical  decision 
making,  it  may  be  appropriate  to  design  an  ADS  experiment  consisting  of  a  mix  of  live,  virtual 
and  constructive  forces.  Getting  the  warfighter  in  the  loop  through  the  live  or  virtual  components 
of  the  technology  can  be  a  better  vehicle  for  examining  human  factors  rather  than  a  constructive 
model. 

2.  When  the  problem  requires  an  M&S  architecture  comprised  of  multiple  models 
interoperating  together  and  the  models  must  be  ran  from  geographically  dispersed  sites. 

3.  When  multiple  processors  are  needed  because  the  problem  is  too  large  to  run  on  a  single 
computer. 
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When  should  ADS  be  used?  notes  (Continued) 


4.  When  the  application  requires  HITL,  such  as  testing  a  Radar  Warning  Receiver 
(RWR)  where  the  aircraft  and  RWR  are  placed  in  an  anechoic  chamber  and  signals  from 
simulated  threats  are  injected  into  the  actual  RWR. 

5.  When  the  application  requires  an  operationally  realistic  scenario  that  cannot  be 
constmcted  from  live  forces  alone.  Typical  applications  include  T&E  of  systems 
operating  against  a  high  density  threat  such  as  found  in  an  Integrated  Air  Defense 
System  (IADS)  and  training  exercises  with  large  numbers  of  forces  including  training  at 
the  JTF-level. 

6.  For  wargaming,  when  it  is  difficult  for  all  of  the  participants  to  be  physically 
present  at  the  common  gaming  facility. 
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2.  Is  ADS  better  or  worse  for  specific  types 
of  analysis  already  being  done? 

•  ADS  may  be  appropriate  when  issues  involve: 

-  HITL  interactions 

-  Information  operations  (IW,  C4ISR) 

•  Inherent  limitations  such  as  repeatability  and  cost 
may  place  ADS  in  a  complementary  role 
supporting  traditional  analysis. 


•  ADS  is  better  for  investigating  decision  making  and  HITL  issues. 

•  ADS  is  better  for  the  investigation  of  information  operations  (information 
warfare  and  C4ISR). 

•  As  the  number  of  human  interactions  in  the  ADS  event  increases,  the 
repeatability  of  the  event  becomes  less  likely  and  the  analysis  becomes  more 
anecdotal. 

ADS  is  better  than  traditional  approaches  to  analysis  when  the  problem  involves  human 
factors  and  we  are  looking  for  "plausible"  rather  than  "expected"  or  "likely"  outcomes. 
Constractive  models  don’t  handle  human  factors  very  well  because  these  models 
typically  do  not  allow  human  errors  to  occur.  Constructive  models  usually  assume  that 
operators  will  make  optimal  decisions.  However,  exercises  and  field  tests  suggest  that 
the  real  world  includes  a  "fog  of  war"  which  leads  to  ambiguous,  false  and  time-late 
information.  In  the  real  world  humans  make  mistakes,  but  we  have  a  hard  time 
accounting  for  this  in  our  models. 
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Is  ADS  better  or  worse  for  specific  types  of  analysis  already  being  done: 

notes  (Continued) 


ADS  is  most  appropriate  when  we  are  performing  exploratory  analysis.  Because  of 
small  sample  sizes  and  problems  of  repeatability,  ADS  is  not  appropriate  when  the 
simulations  are  highly  interactive  and  we  need  to  obtain  statistical  validity.  Examples  of 
inappropriate  applications  include  COEAs  where  we  need  to  make  hundreds  of 
excursion  runs  to  examine  different  scenarios,  different  sets  of  initial  conditions  and 
conduct  sensitivity  analysis  to  identify  the  key  drivers  in  the  problem  and  explore  the 
effects  of  uncertainties  in  the  values  of  these  key  variables. 

Although  many  of  its  proponents  envision  using  an  ADS  approach  early  (prior  to 
milestone  I)  in  the  life-cycle  of  an  acquisition  program,  this  may  not  be  a  productive 
use.  At  this  point  in  its  life-cycle  very  little  is  known  about  the  key  characteristics  of  a 
new  system.  Thus  it  may  be  difficult  to  justify  a  high  fidelity  ADS  approach  to  the 
problem  prior  to  milestone  I.  At  this  stage,  broad  scoping  tools  and  low  fidelity  models 
are  most  appropriate. 
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3.  Does  ADS  allow  analysis  to  address  any 
new,  necessary  analytic  questions? 

•  Information  Operations 

-  IW 

-  C4ISR 

•  Horizontal  technology  integration 


•  ADS  has  potential  for  more  credible  analysis  of  information  operations 
(information  warfare  and  C4ISR). 

•  ADS  has  potential  for  better  understanding  horizontal  technology  integration. 

ADS  would  allow  analysts  to  examine  human  factors  problems  such  as  crew  maiming  in 
naval  ships  or  questions  regarding  how  military  operators  react  to  certain  stimuli  and 
how  they  use  information  in  the  tactical  decision  process.  Thus,  in  system  acquisition, 
ADS  could  be  helpful  in  designing  the  Navy's  new  ship  (surface  combatant  of  the  21st 
century)  to  try  to  reduce  its  crew  size  and  to  analyze  the  requirements  for  new  C4ISR 
systems  that  are  information  intensive. 
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4.  How  does  ADS  fit  with  other  types  of 
analytic  capabilities  and  approaches? 


Results  raise  issues 
and  stimulate  questions 


Traditional 


Model 

Experiment/Test 

Model 


ADS 


Results  raise  issues 
and  stimulate  questions 


•  ADS  can  be  used  to  complement  other  analyses,  with  supportive  results, 
mutual  calibration,  sharing  of  data  and  common  scenarios,  which  allows  for 
more  informed  decisions. 

•  ADS  provides  an  environment  where  hypotheses  developed  through  other 
analytic  means  could  be  tested  and  evaluated. 

The  live  and  virtual  components  of  ADS  could  complement  the  traditional  approach  of 
using  constructive  models  for  analysis.  We  might  use  diese  components  early  in  the 
analysis  process  to  pre-play  the  scenario  to  establish  its  credibility  and  ensure  that  we 
have  accounted  for  all  of  the  critical  variables  in  the  problem.  We  could  look  for 
unanticipated  interactions  that  may  have  been  left  out  of  the  constructive  models  and  to 
help  identify  key  issues  to  focus  the  analysis. 

By  providing  the  human  element,  ADS  runs  using  live  or  virtual  components  tiiat  could 
provide  estimates  of  input  parameters,  such  as  average  response  times,  for  the 
constractive  models. 
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5.  What  are  the  Kmitations  of  current  and 
future  ADS? 


•  Common  architecture 

•  Data  collection/reduction/analysis  tools 

•  Non-repeatability 

-  Human  variance 

-  Network  capacity 

•  Bandwidth 

(continued) 


•  Several  limitations  are  inherent  in  the  distributed  nature  of  ADS. 

•  Many  of  these  can  be  overcome  by  scientific  control  of  experiments,  others  through 
technological  or  software  advances. 

We  don't  have  a  common  set  of  verified  and  validated  models,  data  bases  and  tools  to  support 
analysis.  The  DIS  protocols  are  not  fully  developed  and  will  only  support  a  limited  number  of 
applications.  JWARS  will  not  be  ready  for  several  years. 

As  mentioned  earlier,  the  data  collection,  reduction  and  analysis  tools  available  today  to  support 
ADS  are  inadequate.  Moreover,  insufficient  command  attention  (including  funding)  have  been 
given  to  developing  these  tools. 

Because  of  problems  with  repeatability  and  small  sample  sizes  it  will  be  difficult  to  use  ADS  to 
obtain  statistical  vjdidity  or  to  perform  comprehensive  sensitivity  analysis. 

Many  ADS  applications  are  bandwidth  intensive.  We  need  a  set  of  smart  communications 
protocols  that  minimize  unnecessary  data  transfers. 


31 


5.  What  are  the  limitations  of  current  and 
future  ADS?  (Continued) 

•  Control  of  experimental  variables 

•  Latency 

•  Lead  time 

•  Cost 

•  Training  of  the  analysts/decision  makers 


Many  ADS  applications,  especially  T&E  applications  require  real  time  operations.  The 
latency  associated  with  the  current  DIS  protocols  may  not  support  these  applications. 
Due  to  the  lack  of  an  M&S  infrastracture,  there  is  a  significant  lead  time  and  cost 
associated  with  conducting  an  ADS  experiment.  This  includes  the  time  and  cost  to  build 
the  databases  and  to  configure  the  models  for  the  experiment. 

The  ADS  paradigm  envisions  a  free  flow  of  information  across  the  network  connecting 
requirements  folks,  designers,  developers  and  users.  Issues  include  sharing  of 
proprietary  information  over  the  network,  multi-level  security,  industry's  access-to 
sensitive  information  and  how  many  and  which  particular  concept  design  teams  to  invite 
from  industry;  The  ADS  paradigm  is  designed  to  enable  industry  to  participate  earlier  in 
the  acquisition  process  by  testing  new  concepts  in  common  synthetic  environments 
accessible  by  government  and  industry.  But  the  industry  may  not  be  willing  to 
participate  if  this  means  that  competitors  could  access  proprietary  information  through 
the  network.  By  allowing  one  firm’s  models  to  interact  with  models  from  another  firm, 
they  may  be  able  to  reverse-engineer  the  models  to  learn  a  lot  about  how  the  other  firm 
plans  to  design  their  system.  Similarly,  the  government  will  want  to  limit  access  to 
"black"  programs.  How  will  it  decide  which  firms  to  invite  to  participate  in  the  synthetic 
environment  and  which  to  exclude? 


32 


6.  Are  there  pitfalls  in  analysis  using  ADS? 

•  If  piggy-back  on  training  exercises 

-  Training  objectives  vs.  analytic  objectives 

-  Data  collection  objectives 

•  Reliability  of  network  and  simulations 

•  Failure  to  plan  for  data  coUection 

•  Misinterpretation  of  results 

(Continued) 


•  Training  or  exercise  objectives  usually  take  precedence  over  analysis  objectives. 

•  Lack  of  control  may  invalidate  analytic  results. 

ADS  experiments  that  include  both  training  and  analysis  objectives  may  not  serve  either 
purpose  well.  The  training  and  analysis  objectives  can  sometimes  be  at  odds.  Training 
exercises  are  designed  to  prepare  forces  to  conduct  operations  against  the  threat  most 
likely  to  be  faced  during  the  next  deployment  (six  months).  Acquisition  experiments  on 
the  other  hand,  want  to  test  the  capability  of  a  proposed  system  against  the  threat  that  is 
likely  to  exist  once  the  system  is  fielded.  This  could  be  a  threat  20-30  years  in  the 
future.  The  training  objectives  of  the  exercise  will  tend  to  dominate  and  it  is  unlikely 
that  the  objectives  to  support  the  acquisition  experiment  will  have  much  influence  over 
the  scenario  and  threat  in  place  to  support  training. 

Other  problems  occur  when  mixing  training  with  analysis  objectives.  For  example, 
training  expects  learning  to  occur  between  trials  but  analysis  expects  the  replications  to 
be  independent  trials. 

In  an  ADS  experiment  that  involves  a  human  element,  it  is  very  difficult  to  control  what 
factors  are  held  constant  between  replications.  If  multiple  replications  involve  the  same 
group  of  participants,  the  analysis  must  account  for  the  learning  that  takes  place  finm 
ran  to  ran. 

Extreme  care  is  required  to  develop  hypotheses  for  ADS  experiments  that  will  support 
acquisition  decisions  and  that  are  testable.  In  some  recent  experiences  the  hypotheses 
were  too  broad  and  untestable. 


6.  Are  there  pitfalls  in  analysis  using  ADS? 
(Continued) 

•  Failure  to  understand  assumptions/limitations 

•  Lack  of  planning  to  accommodate  disruptions 

•  Failure  to  correctly  design  experiments 

•  Lack  of  coordination  among  analysts 

•  Lack  of  pre-exercise  training 

•  Lack  of  configuration/version  control 
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7.  Does  the  use  of  ADS  affect  the  utility  of 
analytic  results? 

•  Cause  and  effect  relationships  difficult  to  establish 

•  Easier  to  communicate  results 

•  May  offer  more  credibility  when  analyses  involves 
human  factors 

•  Improve  communication  through  visualization 


•  Results  from  ADS  are  easier  to  communicate  and  sell  due  to  operator  participation  and 
visualization  capabilities 

•  Results  which  indicate  cross-domain  analysis  have  more  credibility. 

•  Credibility  of  cause  and  effect  relationships  can  be  more  questionable  in  ADS. 

Because  of  small  sample  sizes  and  problems  with  repeatability  it  may  be  very  difficult 
to  establish  cause  and  effect  relationships  through  ADS.  It  is  important  to  remember  that 
ADS  experiments  that  are  highly  interactive  will  produce  results  that  may  be  "plausible" 
but  that  are  not  necessarily  "likely"  or  "expected"  results. 

The  high-resolution  graphical  systems  that  allow  visualization  of  three  dimensional 
objects  and  their  interactions  may  make  it  easier  to  communicate  the  results  of  ADS 
experiments  to  senior  decision  makers. 
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8.  Are  there  useful  ways  to  utilize  the  results 
from  ADS-based  analysis,  exercise  and  training 
that  differ  from  conventional  methods? 

•  Accelerate  acquisition  process 

•  Earlier/easier  insights  into  design  trade-offs 

•  Improved  communication  through  visualization 

•  Potential  to  bypass  milestones 


•  ADS  lends  itself  to  the  implementation  of  “smart  product”  concepts  which  could 
revolutionize  the  acquisition  process  allowing  for  better  and  more  complete  designs  in 
less  time  and  fewer  milestone  reviews. 
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9.  Do  ADS-based  analysis  capabilities  make 
VV&A  any  harder,  easier  or  different? 


•  Harder/Impossible 

-  VV&A  federation 

-  Configuration  control 

-  Certification  of  databases 

♦  HITL  may  facilitate  perception  of  validation 


•  W&A  will  be  more  difficult. 

•  The  combination  of  a  number  of  validated  elements  does  not  result  in  a  validated  whole. 

•  Data  certification  will  be  more  difficult. 

•  HITL  will  facilitate  the  perception  of  validation. 

VV&A  of  ADS  systems  will  be  much  more  difficult  than  stand-alone  systems  because  the  models  and 
simulations  that  are  being  linked  together  were  developed  by  different  vendors,  using  different 
technologies  and  were  not  originally  designed  to  work  together. -First  the  individual  models  must  be 
V&V'd  as  stand-alone  tools.  (V&V  of  the  stand-alone  models  is  not  done  well  today.  The 
performance  of  many  modeled  weapon  systems  against  third-world  threats,  in  the  scenarios  and 
operational  environments  postulated  today,  is  significantly  affected  by  phenomena  that  are  poorly 
understood,  much  less  modeled.  This  includes  the  effects  of  multi-path  and  background  noise.) 

After  the  V&V  of  the  stand-alone  models  is  completed,  the  distributed  architecture  must  be  examined 
to  ensure  that  the  individual  pieces  fit  together  as  a  credible  whole.  Thus,  the  models  must  share 
common  assumptions  and  define  variables  to  mean  the  same  thing.  The  outputs  of  one  model  must  be 
in  the  form  expected  of  the  second  model,  if  they  are  to  be  used  as  inputs' to  this  model. 

The  current  litmus  tests  to  establish  the  credibility  of  ADS  for  training  applications  is  whether  Subject 
Matter  Experts  (SMEs)  can  discriminate  the  simulation  from  the  real  system.  This  criteria  is  not 
appropriate  for  acquisition  analyses  that  use  ADS  because  here  the  systems  have  not  yet  been  built 
and  the  SME  have  no  operational  experience.  Thus,  the  models  cannot  be  tested  directly  against  the 
real  world  systems. 
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MORS  Advanced  Distributed  Simulation  for  Analysis  ‘96  Workshop 
Test  and  Evaluation  Working  Group  Report  (WG  3) 


Introduction 

Test  and  Evaluation  (T&E)  presents 
many  opportunities  for  anal5dical  support, 
but  it  is  not  always  in  the  traditional  sense, 
analysis.  In  fact,  a  T&E  can  be  carried  out 
without  any  analysis  being  done.  The 
simplest  situation  would  be  to  try  something 
out  and  see  if  it  does  what  you  want  it  to. 
Unfortunately,  that  is  rarely  the  case,  or 
more  importantly,  rarely  practical  for  T&E 
in  the  Department  of  Defense,  DoD.  The 
considerations  which  drive  us  to  require 
analytic  rigor  in  T&E  are: 

•  Cost  containment  in  T&E 

•  Complexity  of  military  systems 

•  Justifying  the  strengths  and  weaknesses 
of  development  systems 

•  Complexity  of  operational  environments 

•  Achieving  confidence  our  fighting  forces 
will  have  good  capabilities 

•  Justifying  the  dollars  to  be  spent  on 
procurement  and  fielding 

In  this  context,  the  T&E  Working  Group* 
(WG)  addressed  the  nine  issues  posed  by  the 
Workshop  Chairperson  in  the  terms  of 
reference  for  this  Workshop. 


Types  of  T&E  Applications  for  ADS 

Given  the  considerations  above,  the  T&E 
(WG)  addressed  which  types  of  analyses  are 
applicable  to  each  of  the  three  traditional 
phases  of  the  T&E  mission:  T&E  planning, 
test  conduct  and  evaluation.  These  phases 
and  their  associated  analyses  are  not  done 
independently  of  each  other,  and  therefore, 
the  analyses  are  often  not  mutually 
exclusive.  The  T&E  WG  did  find  that  there 
were  some  general  differences  among  the 
analyses  done  in  support  of  the  three  phases. 

For  test  planning,  simulations  are 
typically  used  for  developing  test  conditions 
or  scenarios.  The  main  purpose  is  to  ensure 
that  the  test  will  create  sufficient  events  or 
opportunities  for  the  test  objectives  to  be 
satisfied.  Other  uses  of  simulation  in  the 
planning  phase  include  determining  the 
minimum  resources  needed  to  create  the  test 
conditions,  developing  procedures  for  the 
use  of  the  system  under  test  and  training  the 
operators  for  using  a  new  capability.  The 
pre-test  uses  of  simulation  are  generally 
aimed  at  achieving  efficient  use  of  test 
resources.  These  applications  usually 
require  less  fidelity  and  less  rigor  for 
accreditation  than  do  the  use  of  simulations 
to  support  test  conduct  and  evaluation. 

In  support  of  test  conduct,  simulations 
are  used  to  enhance  the  realism  of  the 
environment  or  to  generate  stimuli  to  the 
system  under  test.  Here  the  key  is  getting 
adequate  realism,  commensurate  with  the 
maturity  of  the  t  est  item,  at  less  cost  than 


*Members  listed  at  the  end  of  this  report 
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would  be  required  for  deploying  live  assets. 
Additionally,  simulations  are  used  to 
overcome  test  limitations  such  as  simulating 
kills  in  combat  (safety)  or  electronic 
countermeasures  (open  air  operations 
security  restrictions  or  lack  of  threat 
systems). 

In  the  evaluation  or  post-test  phase, 
simulations  are  generally  used  to  explore 
conditions  other  than  those  which  occurred 
during  the  test.  The  two  most  common  uses 
are  (1)  relating  tested  system  performance  to 
performance  in  other  scenarios  or 
environments,  and  (2)  estimating  the 
potential  contribution  of  the  system  being 
evaluated  to  larger  scale  (than  could  be 
tested)  mission  accomplishment,  e.g.,  force 
effectiveness  or  campaign  outcomes. 

T&E  Perspective  on  Simulation 

In  reviewing  the  uses  of  simulation  in 
support  of  T&E,  three  principles  surfaced 
often  in  the  WG  discussions.  The  first  is 
that  the  item  under  test,  whether  it  be  a 
component,  the  integrated  system  or  an 
operational  unit  employing  the  system,  must 
be  present  in  a  live  sense.  One  does  not 
have  a  test  of  the  system  if  the  system,  as 
fabricated  and  operated,  is  not  actually 
present  and  operating.  In  contrast, 
simulations  which  also  simulate  the  system 
of  interest  can  be  used  to  test  or  evaluate 
concepts,  system  design  and  procedures  as 
well  as  provide  an  educated  estimate  of  what 
test  conditions  would  be  most  useful  for 
meeting  the  overall  T&E  objectives. 

The  second  recurring  principle  is  that  in  a 
literal  sense,  all  testing  is  simulation.  That 
is,  testing  and  simulations  to  explore  other 
conditions  are  not  real  combat  and  are  not 
the  real  operators.  As  a  minimum,  the  use  of 


a  system  by  its  operations  in  a  test 
environment  is  influenced  by  the  fact  that 
the  operators  are  being  observed  and 
instrumentation  is  interacting  with  a  system 
under  test.  Looking  at  the  T&E  of  military 
systems  in  both  retrospect  and  the  present, 
the  WG  concluded  that  systems  under  test 
always  interact  with  some  combination  of 
live  virtual  and  constructive  environments. 
In  this  context,  T&E  has  been  a  precursor  to 
and  is  a  leader  in  Advanced  Distributed 
Simulation  (ADS). 

The  third  principle  repeatedly  discussed 
by  the  Group  is  that  “caveat  emptor”  applies 
to  the  uses  of  ADS.  The  T&E  community’s 
experience  to  date  is  that  the  wide  variety  of 
“demonstrations”  of  ADS  technologies  have 
not  been  calibrated  or  substantiated  as 
representing  reality.  There  are  few  direct 
comparisons  of  simulations  with  the 
interactions  among  the  systems  and 
environments  that  would  have  occurred  in 
the  actual  mission  conditions  assumed  to  be 
simulated.  When  such  comparisons  have 
been  made,  we  have  learned  that  there  is 
much  work  to  be  done  to  achieve  a  test 
quality  reality  match.  In  other  words,  there 
is  considerable  risk  of  measures  of 
performance  (MOPs)  obtained  from  ADS 
representations  differing,  with  unknown 
biases,  from  the  MOPs  that  would  have  been 
obtained  from  real  operations.  The  T&E 
community  is  very  sensitive  to  this  risk 
because  we  are  usually  held  accountable  for 
the  accuracy  of  our  findings,  and  there  is 
often  evidence  that  is  brought  to  our 
attention  in  the  event  that  we  could  have 
been  wrong  in  our  characterization  of  system 
performance.  Consequently,  the  T&E 
experience  led  the  T&E  practitioners  to 
views  ranging  from  “cautious  optimism”  to 
“skepticism”  on  the  apparent  unbridled 
proposals  for  the  uses  of  ADS  to  create 
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complex  combat  situations  by  linking  many 
disparate  simulators  and  simulations. 

By  raising  these  recurring  principles,  the 
T&E  WG  members  were  on  the  verge  of 
being  gored  by  the  horns  of  a  dilemma  they 
created.  That  dilemma  is  “How  to  be 
proponents  for  the  uses  of  ADS  in  T&E 
while  cautioning  everyone  that  there  are 
many  unknown  or  unquantifiable  risks  in 
doing  so.”  Our  approach  to  reconciling  this 
apparent  contradiction  is  to  encourage  the 
developers  and  users  of  ADS  to  collaborate 
with  the  testers  and  evaluators  whenever 
possible  to  do  rigorous  comparisons  of  the 
results  of  interactions  in  simulations  with 
results  of  interactions  that  occurred  live. 
The  live  sources,  in  order  of  priority,  are 
well  documented  combat  events,  operational 
and  other  system  tests,  and  documented 
training  events.  These  comparisons  are  best 
done  legitimately  in  the  context  of  the 
scientific  method:  use  the  simulation  to 
“predict”  the  interaction  outcomes  for  the 
conditions  of  the  live  interactions;  observe 
and  document  well  the  live  interaction 
outcomes;  and  analyze  the  similarities  and 
differences  between  the  outcomes.  It  is 
important  to  apply  common  sense  and 
military  judgment  in  the  comparative 
analysis  of  simulations  and  live  events.  In 
the  case  of  combat,  it  is  difficult  to  get 
complete  and  accurate  simulations  to  differ 
from  the  combat  situations  they  are  trying  to 
represent.  Generally,  this  approach  to 
improves  simulations  and  builds  confidence 
in  their  records.  In  testing  and  training, 
there  are  often  constraints  which  cause  those 
that  are  live  to  use  over  time  is  called 
“Model-Test-Model”  (M-T-M).  M-T-M  is  a 
partnership  between  those  who  build 
simulations  and  those  who  collect  data  from 
and  create  live  events.  This  partnership  is 
symbiotic  in  the  sense  that  the  two  efforts 
complement  each  other  in  mitigating  their 


respective  limitations.  The  mode  of 
operation  should  be  that  T&E  and 
simulation  are  inseparable;  they  need  each 
other  in  order  for  each  to  gain  credibility. 

Assessment  of  Current  ADS  Applications  to 
T&E 


There  are  generally  two  capabilities,  under 
the  umbrella  of  ADS,  which  have  been  used 
to  date  in  T&E.  The  first  is  DIS  compliant 
systems  such  as  SIMNET  and  its  recent 
upgrades.  The  second  is  an  ad  hoc 
integration  of  simulations  to  drive  test 
conditions  or  scenario  inputs.  Examples  are 
Aggregate  Level  Simulation  Protocol 
integration  of  simulations  to  drive  C4ISR 
(Connnand,  Control,  Communication, 
Computer,  Intelligence,  Surveillance  and 
Reconnaissance)  test  and  training  exercises 
and  scenario  or  event  scripts  driving  test 
devices  such  as  anechoic  chambers.  The 
following  table  lists  some  strengths  and 
weaknesses  of  the  current  users  of  ADS  in 
T&E. 
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Table  1:  Ciorrent  ADS  Considerations  for  T&E 


PROS  CONS 

GENERAL;  o  Fast  set-up  o  Investments 

o  Process  visibility  (lessons  learned)  o  Semantic  compatibility 

o  Low  fidelity 
o  Requires  W&A 


PLANNING:  o  Develop  T«&E  issues  and  measures  o  Competes  for  access  to 

0  Refine  concept  of  operations  and  simulations, 

procedures  networks  and 

o  Train  operators,  rehearse  warfighters 

CONDUCT:  o  C4ISR  scenario  drivers  o 

o  Emulate,  simulate  and  stimulate  to 

reduce  live  assets  o 

o  Interoperability/Joint  Operations  o 


Simulation  incompatibilities 
(e.g.,  terrain,  detection) 
Operations  “noise/friction” 
Updates  of  various 
simulations  based  on 
test  item  changes  or 
changes  in  interacting 
simulations 


EVALUATION:  o  End  to  end  assessments  o  Replacing  too  much  test 

o  Sensitivity  analysis  with  other  simulations 

o  Evaluate  untestable  o  Performance  measures 

conditions  biased  by  unknown 

simulation  flaws 


The  WG  noted  that,  in  terms  of  personnel.  New  Application  for  ADS  in  T&E 
dollars  and  coordination  costs,  DIS  is 

generally  more  costly  than  using  The  WG  identified  two  areas  in  which  ADS 

constructive  simulations,  as  was  done  in  the  would  enable  new  or  significantly  improved 

past.  The  greatest  benefits  to  date  from  DIS  capabilities  in  support  of  T&E.  The  first  is 

have  been  better  preparations  of  the  test  unit  the  general  use  of  a  common,  synthetic 

for  operational  tests.  battlespace  to  tie  requirements  generation  to 

analysis  of  alternatives  and  in  turn  to 
simulations  of  test  scenarios.  With  such  a 
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common  synthetic  battlespace,  one  can 
relate  test  findings  to  the  system  attributes 
justifying  program  approval.  The  second 
type  of  new  capability  is  to  introduce  high 
fidelity  interactions  with  other  systems 
earlier  in  the  development  process.  This  can 
enable  a  complete  record  of  all  interactions 
with  the  system  under  test  by  means  of 
recording  all  protocol  data  units  exchanged 
in  a  complex  simulated  combat.  In  the  case 


where  the  system  failed  to  perform  a 
required  function,  this  record  can  be  used  to 
replay  the  conditions  which  led  to  the  failure 
as  a  means  of  checking  the  efficacy  of  a  fix 
to  the  system.  The  new,  necessary  T&E 
questions  that  the  WG  identified  which  ADS 
could  help  address  are  listed  in  the  table 
below. 


Table  2:  New,  Necessary  T&E  Applications  for  ADS 

GENERAL: 

o  ADS  allows  common  synthetic  battlespace,  enabling  several  benefits 

PLANNING: 

0  Effectiveness  analysis  in  more  realistic  environment,  common  with 
DT/OT 

0  Requirements  traceability  through  test  cases 
o  Adaptive  learning  addressed  during  test  design 

CONDUCT: 

o  Introduction  of  interactions  with  high  fidelity  assets  sooner  in 
development 

o  Application  of  effects  of  systems  without  “outdoor  testing”  (e.g.,  ECM) 
o  Overcome  more  test  limitations  (e.g.,  safety,  availability  of  assets, 
environmental  conditions,  etc.) 

EVALUATION: 

0  Recording  capabilities  for  “re-creation  of  test  conditions”  (check  out 
fixes  to  problems) 

0  Enhancement  of  DT  evaluation  to  operational  consequences 

0  Enhancement  of  total  system  performance  evaluations 

0  Potential  of  evaluating  how  well  battle  damage  assessments  are 
performed 

o  Multi-media/visual  aids  as  analysis  tool  —  spinoff  to  training  and 
operations 

-  Replay  battles  to  gain  insights  about  outcomes 
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ADS  Fit  With  Other  Analytical  Capabilities 

As  discussed  above,  the  T&E  WG  felt 
that  the  live  aspects  of  ADS  simulations  can 
complement  other  types  of  simulations  and 
interactions  among  simulations  by  providing 
data  to  support  validation  and  accrediting 
various  ADS  configurations.  Also  exploring 
larger  or  different  scenarios  or  combat 
environments  with  ADS  can  help  relate 
observed  system  performance  to  more 
abstract  notions  of  military  worth.  The  final 
topic  the  WG  considered  in  this  area  was  a 
clear  distinction  between  compliance, 
compatibility  and  interoperability  in  the 
context  of  VV&A.  Compliance  is  an 
element  of  verification  in  that  it  is  a 


determination  that  the  information 
exchanged  adheres  to  the  specified  standards 
for  the  exchange.  Compatibility  is  an 
element  of  logical  verification  in  that  it 
addresses  the  consistency  factors  such  as 
resolution,  environmental  fidelity  and  the 
ability  of  the  simulated  entities  to  use  the 
exchanged  data  as  intended.  Interoperability 
is  the  most  difficult  requirement  for 
supporting  validation  and  accreditation.  It  is 
an  indicator  of  how  well  the  simulation 
matches  how  the  actual  objects  behave  in  the 
real  world.  The  following  table  lists  the 
ADS  relationship  with  other  analytical 
capabilities  from  the  T&E  perspectives. 


Table  3:  ADS  Fit  with  Other  T&E  Elements 


COMPLEMENTARY: 

0  Validate  physical  and  behavioral  models  in  constructive 
simulations 

o  Prepare  for  live  simulations 
o  Compose  simulations  from  existing  models 

0  Running  consistent  simulations  independently  to  address 
elements  of  evaluation 

SYNERGISTIC: 

0  Expand  scope  of  analysis  such  as  scenario  size  and 
environmental  factors 

Fit  and  linking  LIVE,  VIRTUAL  and  CONSTRUCTIVE  simulation  depends  upon: 
o  Compliance  -  Can  be  solved 

o  Compatibility  -  Uncertain  (e.g.,  environmental  fidelity,  resolution...) 
o  Interoperability  -  Tradeoffs,  extensive  validation  tests,  upgrades  to  live 
real  time  assessments. 
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T  iTnitatinns  of  ADS  for  T&E  Organized  by  two  notional  timeframes  on  the 

top  and  the  applicable  phases  of  T&E  on  the 

The  table  below  lists  the  WG’s 
assessment  of  ADS  limitations  for  T&E 


Table  4;  ADS  Limitations 


CURRENT 

FUTURE 

PLANNING: 

0  Plug  &  Play”  capability  for 

system  and  software 

0  W&A  and  M&S  repository 

CONDUCT  AND 

o  Fidelity 

EVALUATION: 

0  Security 

0  Proprietary  M&S  access 

0  Proprietary  M&S  access 

0  Technology  (e.g.  latency) 

o  Technology  (e.g.  latency) 

ALL: 

o  Cost  and  resources 

0  Robust  environmental  M&S 

o  Robust  environmental  M&S 

0  M&S  interoperability 

o  M&S  interoperability  and 

0  W&A 

compatibility 

Two  areas  warrant  further  explanation 
under  this  topic.  The  first  is  the  WG’s 
concern  about  the  notion  of  “Plug  and  Play.” 
Specifically,  the  Group’s  concern  is  that 
complex  configurations  of  simulations  can 
be  constructed  and  operated  with  only 
compliance  being  satisfied.  Especially  for 
large  ADS  federations,  compatibility  and 
true  interoperability  will  be  difficult  or 
nearly  impossible  to  confirm,  and  therefore, 
may  not  be  checked.  The  consequence 
would  be  unknown  biases  and  errors  in  the 
performance  or  effectiveness  measures 
produced  by  the  running  of  such 
simulations. 


The  second  concern  in  Table  4  is  the  cost 
and  human  resources  required  to  develop, 
operate  and  accredit  ADS  elements  and 
configurations.  Although  ADS  as  a  tool 
may  be  easily  reconfigurable,  other  costs  and 
risks  may  make  it  impractical  for  some 
potential  T&E  applications.  The  points  on 
cost  which  were  raised  and  discussed  in  the 
WG  are  summarized  in  the  following 
paragraph. 

Good  models  or  simulations  usually 
come  with  substantial  price  tags.  They  are 
most  useful  for  mitigating  program  risk  in 
the  areas  of  requirements  generation,  system 
design  and  development  of  employment 
procedures.  T&E  is  the  beneficiary  when 
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models  and  simulation  required  for  system 
development  can  be  accredited  for  reuse  in 
making  efficient  use  of  T&E  resources,  and 
in  examining  conditions  which  could  not  be 
tested.  A  concern  is  that  further  investment 
in  modeling  and  simulation,  solely  for  the 
purpose  of  evaluating  test  findings,  does  not 
always  have  a  good  or  known  return  on 
investment.  We  need  to  develop  a 
methodology  for  determining  trade-offs 
between  spending  dollars  on  simulations 
versus  on  additional  test  capabilities.  There 


may  be  cases  where  it  costs  less  or  we  learn 
more  from  a  test  capability  than  we  could 
achieve  from  spending  on  a  yet  to  be  proven, 
simulation. 

Pitfalls  in  ADS  Application  to  T&E 

From  a  T&E  perspective,  some  of  the 
perceived  pitfalls  in  using  ADS  are  listed  in 
the  following  table. 


Table  5:  ADS  Pitfalls  in  T&E 

PLANNING: 

o  Reluctance  to  give  up  trasted  methods  for  unproven  techniques 

PLANNING/: 

CONDUCT 

0  Fidelity/scaleability  including  model  approach,  characteristics 
and  analyst’s  knowledge;  and  overconfidence  in  “visual  displays” 
0  VV&A  including  time,  responsiveness,  cost,  complexity  and 
configuration  management 

CONDUCT: 

o  Latency  issues  &  impact  on  areas  such  as  ECM,  ACM,  C3I 

0  Interoperability  disconnects  with  LIVE,  VIRTUAL  and 
CONSTRUCTIVE  entities  including  reliability,  scheduling, 
seamless  issues  and  reality  “mismatches” 

EVALUATION: 

0  Due  to  disparate  sites,  difficult  to  maintain  positive  control  over 
input  data  and  data  collection  and  reduction 

Most  of  these  issues  have  been  addressed 
already.  Two  are  worthy  of  explanation 
here.  One  is  infatuation  with  visualization. 
The  T&E  WG  offers  two  cautions  here.  The 
first  is  that  with  the  technology  advances 
which  enable  realistic  looking  imagery,  we 
may  be  generating  imagery  which  is  not 
required  for  analysis.  Typically,  imagery 
will  be  necessary  for  combat  fought  with 
manned  simulators.  This  leads  to  the  second 
caution  for  visualization;  viz.  when  we  see 


realistic  looking  imagery,  we  forget  to 
question  whether  or  not  the  underlying  data 
and  algorithms  are  faithfully  representing  the 
intended  real  phenomena.  Nice  looking, 
plausible  responses  are  not  sufficient  to 
assure  that  the  interactions  are  correct. 

The  second  issue  in  Table  5  requires 
some  additional  explanation  on  the  topic  of 
latency.  As  ADS  federations  pass 
information  over  heavily  loaded  or  wide  area 
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networks,  the  arrival  and  processing  times  at 
different  entities  will  likely  not  match  the 
times  that  the  represented  influences  would 
be  realized  in  the  real  world  interactions. 
For  phenomena  such  as  those  relevant  to 
electronic  combat  where  complex 
waveforms  are  traveling  and  interacting  at 
the  speed  of  light,  the  latency  problems 
inherent  in  ADS  may  be  a  fatal  pitfall  for 


those  applications. 

ADS  Effects  on  the  Utility  of  Analytical 
Results 

The  T&E  WG’s  assessment  of  strength 
and  weaknesses  of  ADS  effects  on  the  utility 
of  analytic  results  are  listed  below. 


Table  6:  ADS  Effects  on  T&E  Analysis 

IMPROVEMENTS: 

o  Quality  of  results 

-  Tailorable  scope 

-  Prepare  for  live  tests 

-  Model  reuse 

0  Efficiency 

-  Get  system  requirements,  design,  operational  test 
measures  consistently  aligned. 

-  Quickly  reconfigure  simulations 

-  Analyze  more  issues 

CHALLENGES: 

0  Transition  period 

0  Non-compliance 

0  Integrating  legacy  models 
o  Many  configurations  to  accredit 

0  Simulation  runs  often  not  repeatable 

New  Wav  to  Utilize  the  Results  from  ADS- 
based  Analysis.  Exercise  and  Training 

The  T&E  WG  considered  this  issue  in 
terms  of  a  paradigm  shift.  The  old 
paradigms  that  segregated  users  —  analysts, 
testers,  trainers  and  mission  planners  —  and 
aligned  event  phases  in  chronological  order 
—  planning,  conduct  and  evaluation  —  must 


be  broken  if  the  trae  benefits  of  ADS  are  to 
be  derived.  This  “paradigm  breaking” 
would  be  the  direct  result  of  achieving  the 
ADS  vision  of  a  simulation  toolkit  that  can 
be  tailored  for  each  event,  which  enables 
portrayal  of  a  continuous  war  and  which 
leverages  the  insights  and  experiences  of  an 
entire  community  of  ADS  users  and 
developers.  While  ADS  is  not  a  panacea,  it 
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does  offer  a  vision  and  new  way  of  thinking  improvements  in  the  three  major  categories 

about  how  the  DoD  does  its  business.  The  depicted  in  the  following  table. 

ADS  technologies  which  could  lead  to  a 
common  synthetic  battlespace  should  bring 


Table  7:  Key  Concepts  Supporting  New  Uses  of  ADS  in  T&E 


o  Visualization  in  three  dimensions 

-  During  and  after 

-  Midcourse  corrections 

-  Improved  test  control 

-  Pure  and  player’s  view 

-  After  Action  Review  (AAR)  -  multimedia,  operational  impact 

o  Expand  the  scale  and  scope 

-  Broader  issues 

-  Larger  common  data  set  that  leverages  training  products  for  T  &E 

-  Increase  in  number  of  data  sets 

-  Perform  “what  if’  analyses 

o  Integrate  across  functions  and  categories 

-  Non-linear  merging  of  events  (test,  training  and  analysis),  simulations  (LIVE,  VIRTUAL  and 
CONSTRUCTIVE)  and  phases  (planning,  conduct  and  evaluation) 

-  Balance  use  of  tools  based  on  feedback 

-  Requirements  and  baseline  tracking  for  continuous  assessment,  adaptive  sequencing 


The  radical  shift  proposed  by  ADS  for 
simulation  methodology  and  technology 
makes  it  difficult  to  fully  fathom  the  benefits 
that  will  be  offered  by  its  employment  or  to 
appreciate  just  how  muddy  the  lines  between 
traditionally  distinguishable  users  and  events 
will  become.  The  Joint  Advanced 
Distributed  Simulation  (IADS)  Joint  Test 
Force  is  forging  an  excellent  first  step  in 
seeing  how  the  vision  can  become  a  reality; 
however,  exploration  and  feedback  from  the 
use  of  ADS  must  not  stop  there.  Many  users 
of  simulations  are  already  probing  into  the 
vision  of  ADS,  even  if  not  using  specific 
DIS  or  High  Level  Architecture  (HLA) 


standards  and  protocols.  Their  lessons 
learned  must  be  examined  for  applicability 
to  ADS.  The  DoD  should  also  consider 
augmenting  some  events  (test,  training  or 
analysis)  financially  and  technically  to 
establish  them  as  test  cases  for  proving  out 
ADS.  This  seed  money  and  intellectual 
support  would  enable  users  to  explore  new 
avenues  while  still  insuring  that  the 
requirements  which  are  driving  their  use  of  a 
simulation  or  suite  of  simulations  is 
satisfied. 

Verification.  Validation  and  Accreditation 
fW&A)  Considerations 
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The  WG  identified  aspects  of  VV&A  for 
ADS  that  are  envisioned  to  be  easier,  harder 


and  different  from  our  current  W&A 
experiences  as  shown  in  the  Table  8. 


Table  8:  VV&A  Considerations 


o  Use  of  commonly  accepted, 
recommended  practices 
o  Use  of  W&A  histories  from 
central  repository 
o  Previously  validated  pieces 
o  Reuse  of  previous  VV&A 
reduces  cost 


o  Distribution  of  cost  and 
schedule  across  programs 
o  Validation  across  different 
levels  of  fidelity 
o  Dependence  on  successful 
data  collection  (time, 
resources  and  cost 
complexity) 

o  Configuration  management 
issues 


o  W&A  not  a  “cure 
all”  —  analyst  must 
interpret  results  in 
much  more  complex 
environments 


For  the  items  in  the  “easier”  column  to  be 
realized,  it  is  essential  that  an  accessible, 
complete  and  user  friendly  repository  of 
VV&A  records  and  histories  be  kept  current. 
By  far,  the  most  difficult  aspect  of  VV&A 
for  ADS  identified  by  the  WG  is  generating 
validation  data  for  and  accrediting  the 
“interoperability”  among  the  unbounded 
combination  of  simulation  interactions  for 
each  ADS  configuration  and  application. 


The  Group  noted  that  generally  less 
stringent  accreditation  standards  would 
apply  to  test  planning  applications  than 
would  apply  to  test  conduct  applications. 
Depending  on  the  use  of  simulation  - 
generated  measures,  accreditation  criteria 
could  ran  the  whole  range  of  rigor  when  the 
application  is  for  evaluation.  The  WG 
discussed  the  collective  concern  of 
practitioners’  not  doing  accreditation 
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because  the  construction  and  running  of  an 
ADS  is  likely  to  take  less  effort  that  would 
be  required  to  accredit  that  ADS 
configuration.  This  is  of  most  concern  for 
applications  in  test  conduct  and  mission 
rehearsal  where  accuracy  of  simulated 
processes  is  especially  important. 

Common  Themes  from  the  T&E  WG 
Deliberations 

The  points  that  tended  to  recur 
throughout  the  WG  deliberations  are 
summarized  below.  The  “o,”  and 
signify  negative,  neutral  and  positive  points 
respectively. 

“Reality  matching” 
interoperability,  much 
more  difficult  than 
“plug  &  play” 
compliance 

Biases  in  test  item 
performance  measures 
resulting  from  ADS 
unknowns 

o  Visualization 

+Causality  for 
lessons  learned 

-  Promotes  acceptance 
without  knowing 
how  ADS  works 

o  “Testing”  and  “simulation” 

(pre  ADS  notions)  need  each 
other:  through  M-T-M 
testing  and  other  simulations 
complement  each  other  in 
overcoming  limitations 

o  VV&A 

+  Reuse 

-  Configuration 


control/complexity 
+  Potential  for 
common  synthetic 
environment  to 
link  requirements, 
development, 
techniques 

of  use,  T&E,  training 
and  operations 
+  Broader  scope/ 
“untestables” 

+  More  versatility 

T&E  WG  Recommendations 

The  T&E  WG  recommendations  for  ADS 

‘96  to: 

o  The  HLA  include  data 
collection  and  recording 
standards 

o  Develop  standards  for 
“interoperability” : 
semantic  or  reality 
matching 

o  Establish  resource 

repository  requirements 
and  standards  (to  include 
lessons  learned) 

o  Promote  awareness  to 
dispel 

-  “Plug  &  play”  myth 

-  Visualization  = 

“must  be  true” 

o  Initiate  research  on  metrics 
for  alerting  ADS  potential 
problem  or  flaws 

o  Require  ADS  ports  or  hooks  in 
operational  systems 
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INTRODUCTION/GROUP  MEMBERSHIP  -  Slide  2 

In  preparation  for  the  Training,  Mission  Rehearsal  and  Alternative 
Courses  of  Action  Working  Group  (WG4),  an  attempt  was  made  to  assemble 
representatives  from  both  the  training  and  analysis  communities.  A  deliberate  effort  was 
made  to  include  the  “user  community”  as  described  by  the  new  emphasis  on  joint  and 
interoperability  training.  The  resulting  attendance  at  the  WG  precipitated  lively 
discussion  from  both  the  trainers/operators  and  the  analysts.  We  appreciated  the 
participation  of  the  WG  membership,  which  provided  the  opportunity  to  form  a  bridge 
between  two  very  separate  communities.  One  member  of  the  group  used  the  analogy  that 
the  training  and  analysis  practitioners  are  residents  of  two  different  planets  with  the 
trainers  from  Mars  and  the  analysts  from  Venus.  The  thought  provoking  dialogue  that 
occurred  in  this  session  has  resulted  in  recommendations  that  will  assist  in  bridging  the 
gap  between  the  two  worlds. 

CAPT  Drew  Beasley,  Program  Manager  of  the  Joint  Simulation  System 
(JSIMS)  Program  was  co-chair  of  WG4  and  provided  an  excellent  overview  that  helped  to 
tie-in  with  the  current  efforts  that  are  underway  to  build  a  simulation  tool  for  training  and 
training  analysis  at  a  full  range  of  training  activities.  COL  Tom  Verbeck  heads  the 
Technology  Division  of  the  Joint  Warfighting  Center  (JWFC)  which  is  responsible  for 
working  with  the  CINCs  to  translate  their  training  requirements  into  the  framework  for 
JSIMS.  The  technical  community  and  specifically  Defense  Modeling  and  Simulation 
Office  (DMSO)  was  represented  by  Dr.  Judith  Dahmann.  Other  members  of  the  WG 
were  from  either  Joint  and  Service  Staffs,  FFRDCs  or  supporting  industry  contractors 
involved  in  the  two  communities. 

In  order  to  set  the  stage  for  meaningful  discussion  between  the  training 
and  analysis  communities  a  portion  of  the  first  day  was  set  aside  to  receive  relevant 
presentations  on  the  JSIMS  program,  technology  issues,  the  role  of  the  JWFC  and  an 
example  of  distributed  models  from  the  functional  area  for  logistics  support. 


Group  Process/Agenda 


•  BACKGROUND 

-  ADS  For  Training 

-  Venn  Diagram 

-  Current  Initiatives 

-  Methodology 

•  PRESENTATIONS 

-  Joint  Simulation  Initiatives,  CAPT  Beasiey 

-  Technology  Issues,  Dr  Dahmann 

-  Joint  Warfighting  Center,  COL  Verbeck 

-  Distributed  Intelligent  Architecture  For  Log,  Ms  Moore 

•  DISCUSSION 

-  DoD  Definition 

-  Context  Of  Problems 

-  Treatment  Of  Issues 

-  Framework  For  Analysis 

«  RECOMMENDATIONS 


GROUP  PROCESS/AGENDA  -  Slide  3 

By  way  of  background,  an  opening  discussion  session  was  held  to  introduce 
individuals  to  each  other  and  to  the  perceptions  of  ADS,  as  it  has  been  applied  in  the 
training  area.  The  discussion  produced  the  diagram  and  information  found  on  Slide  4. 

Relevant  current  initiatives  and  methodologies  were  covered  by  the 
briefings/presentations  and  discussions  during  the  first  day’s  session. 

A  JSIMS  overview  briefing  presented  by  CAPT  Beasley  covered  the  background, 
objectives  and  development  concept  and  a  graphic  of  how  JSIMS  fits  into  the  range  of 
modeling  and  simulation  and  addressed  our  theme  regarding  the  “Requirements  for 
Analysis  within  the  Training  Domain”.  The  questions  raised  and  discussed,  included: 

•  What  analytic  tools,  techniques,  and  procedures  can/should  the 
analysis  community  offer  to  the  trainers? 

•  What  are  the  critical  elements  for  analysis/support  for  exercise 
planning?  For  after  action  review?  For  supporting  training  objectives? 

•  How  can  analysis  and  ADS  be  used  in  support  of:  training  events, 
trainees  performance,  tactics  /doctrine,  weapons  systems  and  support 
systems  development,  force  structures  and  operational  planning? 

Dr.  Judith  Dahmann,  DMSO,  gave  an  excellent  summary  of  technology  issues 
concentrating  on  the  HLA  and  run  time  infrastructure  with  further  discussion  of  important 
issues  such  as  aggregation  /disaggregation,  VV&A  and  logical  consistency. 

The  overview  of  JWFC  was  presented  by  COL  Tom  Verbeck.  He  discussed  their 
role  in  working  with  the  CDSfCs  and  their  staffs  to  move  from  their  Joint  Mission  Essential 
Task  Lists  (JMETL)  to  requirements  that  will  feed  JSIMS.  The  JWFC  is  the  “keeper  of 
the  keys”  for  the  Executive  Agents  in  JSIMS.  The  discussion  led  to  a  wide  range  of  ideas 
regarding  support  for  the  joint  training  environment. 


NOTES  Slide  3  (Continued) 

Ms.  Miranda  Moore  of  the  Logistics  Integration  Agency  discussed  an  example  of  a 
distributed  approach  for  improving  the  joint  analytic  logistics  decision  process.  The 
technical  approach  referred  to  as  the  “Distributed  Intelligent  Architecture  for  Logistics 
(DIAL)”,  will  allow  for  existing  functional  area  models  to  interface  and  communicate  to 
cover  logistics  planning  and  integrate  mobilization  with  deployment. 

Early  discussion  by  the  group  indicated  an  unwillingness  to  take  the  time  to  debate  the 
basic  definition  of  ADS,  or  what  it  is  or  isn't  in  great  detail,  but  rather  to  move  directly  into 
the  analysis  and  application  of  ADS  in  the  training  environment.  A  common  treatment  for 
issue  discussion  was  the  general  session  and  introductory  presentations.  This  set  the 
foundation  and  framework  for  the  following  analysis  of  the  WG. 


SCOPE  OF  DEBATE  -  Slide  4 

In  the  past,  there  was  a  relatively  common  interface  between  the  training  and  ADS 
portions  of  this  diagram.  The  U.S.  military  has  developed  an  impressive  array  of 
simulators  and  training  systems  over  many  years.  These  systems,  largely  stand  alone 
single  system  simulators,  became  very  effective  and  widely  used  for  training  individuals 
or  small  teams  (crews)  to  accomplish  their  specific  duties.  These  simulators  range  from 
simple  devices  (such  as  rifle  marksmanship  trainers)  to  more  complex  devices  (such  as 
tank  gunnery  simulators  or  sophisticated  flight  simulators).  The  current  inventory  of 
simulation  systems  does  not  offer  the  opportunity  for  training  in  combined  operations  or 
the  wide  range  of  scenarios  that  will  be  needed  in  joint  and  interoperability  training 
required  by  today’s  world  situation.  The  concept  of  networked  simulations  was  first 
developed  in  the  SIMNET  program,  which  was  sponsored  by  the  Defense  Advanced 
Research  Projects  Agency  (DARPA).  The  linking  of  platform-level  simulators  provides 
the  capability  to  train  as  teams  and  to  set  up  battlefield  scenarios.  This  system  of 
networked,  interactive  simulations  forms  the  intersection  between  training  and  ADS  as 
depicted  in  the  Venn  diagram.  With  the  improvements  in  ADS  we  have  the  opportunity 
to  incorporate  real  time  interaction  during  the  course  of  analysis.  Changes  can  be  made 
in  the  course  of  an  exercise  or  event  that  will  affect  the  outcome,  providing  a  strong 
analysis  tool. 

It  is  important  to  consider  analysis  in  the  support  of  training  and,  in  particular,  in 
support  of  distributed  simulations  used  for  training.  Constructive  simulation  combines 
simulated  combat  forces  and  simulated  systems.  Constructive  simulations  are  widely 
used  for  analysis,  but  they  currently  have  a  relatively  narrow  niche  in  training 
applications.  The  primary  training  value  of  constructive  simulations  has  been  in 
preparing  command  staffs  for  joint  and  service  operations.  Each  of  the  Services  has 
developed  constructive  simulations  to  meet  its  command  staff  training  needs,  but  there 
are  limited  models  available  to  support  joint  training.  The  use  of  constructive  simulation 
has  allowed  the  partial  substitution  for  actual  troop  deployments  or  weapons  systems 
saving  valuable  resources  while  enhancing  training. 
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NOTES  Slide  4  (Continued) 

The  joint  training  environment  will  be  dri' :  t  the  need  for  the  marriage  of  analysis,  training 
and  advanced  distributed  simulations.  Wi  joint  training  leads  the  requirements,  the 
Services  have  specific  activities  which  wii  r  f  riefit  from  support  from  analysis.  The 
SIMNETs  are  in  use  at  Forts  Knox,  Rucker  and  Benning,  Gowan  Field,  Camp  McCain  and 
USAREUR.  The  Navy  uses  Battle  Force  Tactical  Trainor  (BFTT)  for  training  exercises  and 
the  Army  sponsors  ALSP  (Aggregate  Level  Simulation  Protocol)  exercises  like  “Prairie 
Warrior.”  The  analysis  community  must  learn  more  about  training  in  order  to  help  the 
trainers  with  their  problems. 


ISSUE  Analysis  Schema 


•  NAME  OF  ISSUE: 

-  How  is  the  issue  to  be  denoted? 

•  OBSERVATIONS: 

-  What  is  true,  as  a  matter  of  fact  and  pertinent? 

•  CONCLUSION/FINDINGS: 

-  What  further  inferences  or  judgments  can 
reasonably  be  made? 

•  RECOMMENDATIONS: 

-  Who  should  do  what,  when,  how  and  to  what 
effect? 


ISSUE  ANALYSIS  SCHEMA  -  Slide  5 

As  the  group  discussion  progressed  from  the  opening  session  and  briefings,  it  was 
necessary  to  develop  a  methodology  for  the  structured  analysis  of  the  issues  which  were 
surfaced.  The  slide  above  depicts  the  standard  manner  in  which  the  remainder  of  the 
briefing  has  been  formatted.  As  issues  were  brought  up  for  discussion,  they  were 
described  in  the  training/ADS/analysis  context.  The  WG  then  made  observations  and 
developed  conclusions  or  findings  relative  to  the  specific  issue,  before  moving  to  make 
recommendations. 


Issue:  MOEs  for  Training 


•  OBSERVATIONS; 

-  Cost/benefit  of  approaches  (traditional  and  ADS) 
should  be  determined 

-  Tools  available  drive  training 

•  CONCLUSION/FINDINGS: 

-  Training  audience  and  objectives  should 
determine  which  tools  are  used 

-  Less  costly  training  may  support  requirements 

•  RECOMMENDATIONS: 

-  Develop  MOEs  for  evaluating  training 

-  Estabiish  quantitative  measures 


ISSUE;  MOEs  for  TRAINING  -  Slide  6 

It  was  observed  that  additional  Measures  of  Effectiveness  (MOEs)  and  Measures  of 
Performance  (MOPs)  are  needed  for  evaluating  training  effectiveness.  In  many 
instances  the  available  tools  drive  the  training  events.  It  was  also  felt  that  the  cost/ 
benefit  of  traditional  training  approaches  should  be  determined  to  serve  as  a  benchmark 
for  trading  off  the  legacy  systems  for  more  advanced  training  systems.  The  Army’s 
TRADOC  Analysis  Command  (TRAC)  has  been  conducting  training  effectiveness 
analysis  to  evaluate  the  value  of  training.  This  type  of  evaluation  offers  potential  in  the 
ADS  environment.  Additionally,  TRAC  has  developed  a  training  mix  model  to  assist  in 
determining  the  best  combination  of  live,  virtual  and  constructive  based  training. 

The  training  audience  and  the  purpose  or  objective  of  the  training  should  drive  the 
selection  of  the  appropriate  training  tools.  Just  because  it  is  possible  to  link  platform 
level  simulators  to  participate  in  joint  training  events,  is  it  logical  and  desirable  to  do  so? 
It  is  possible  that  simple,  less  costly  training  may  actually  support  the  training 
requirements  of  a  given  category  of  training  adequately,  or  even  better  than  some 
proposed  complex  systems? 

It  was  felt  that  the  training  community  should  develop  additional  MOEs  for 
evaluating  training  effectiveness  and  to  establish  quantitative  measures  that  would  be 
useful  in  determining  the  cost^eneflt  of  traditional  versus  more  advanced  automated 
systems.  The  training  community  needs  embedded  training  feedback  and  evaluation 
tools  to  evaluate  joint/command  staff  decisions.  Resident  tools  are  normally  preferred  to 
shared  tools  and  there  is  a  recognized  strong  need  for  the  CINCs  models  to  be 
interconnected  among  the  other  users/players. 


Issue:  Lack  of  Functional  Modules  to 
Support  Training 

•  OBSERVATIONS: 

-  Post  Cold  War  emphasis  on  logistics...  heightened 
awareness...  seldom  played  in  ADS 

-  Exercises  too  short  to  encompass  major  resupply 
and  IB 

-  Needs  include  intelligence  interaction,  IW,  IB,  C4I 
interfaces,  mine/counter  mine, ... 

•  CONCLUSION/FINDiNGS: 

-  Incorporate  log  requirements  for  training  and 
analysis 

-  IB  links  provide  realism  for  sustainment 

•  RECOMMENDATION: 

-  Include  log  and  C4I  effects  in  federations _ 


ISSUE:  Lack  of  Functional  Modules  to  Support  Training  -  Slide  7 

Just  as  the  post  cold  war  period  and  Goldwater/Nichols  has  changed  the  way  we 
fight  our  wars,  it  has  also  changed  the  way  we  provide  support  for  our  forces.  There  has 
been  a  significant  heightening  of  the  emphasis  on  logistics.  The  functional  support  areas 
are  seldom  played  in  an  ADS  environment,  even  with  the  increased  awareness  of  their 
need  in  today’s  military  forces.  The  cycle  time  for  most  exercises  is  too  short  to 
encompass  the  full  spectrum  of  logistics  support,  let  alone  the  Industrial  Base  (IB).  The 
combat  cycle  at  tactical  combat  level  is  usually  24  hours  or  less.  As  the  level  of  combat 
activity  increases,  the  time  becomes  even  more  compressed  at  lower,  tactical  units.  The 
cycle  time  for  aircraft  or  tanks  engaged  in  combat  for  instance  can  be  minutes  or  even 
seconds.  In  order  to  get  full  training  benefit,  it  may  be  desirable  to  train  at  greater  than 
real  time  for  intense  tactical  combat  activity.  Conversely,  at  the  Joint  Task  Force  or 
CINCs  staff,  the  strategic  level  of  combat  extends  over  greater  periods  of  time.  If  an 
exercise  is  conducted  over  a  period  of  days  or  even  weeks,  the  full  play  combat  support 
does  not  occur  due  to  the  increased  cycle  time. 

In  addition  to  logistics,  other  functional  areas  for  incorporation  include  intelligence 
and  information  warfare,  C4I  interfaces,  IB,  mine/countermine  and  Military  Operations 
Other  than  War  (MOOTW).  The  need  is  evident  for  the  incorporation  of  logistical 
support  requirements  into  models  and  simulations  for  both  training  and  analysis.  It  may 
also  be  necessary  to  modify  the  manner  in  which  the  above  functional  areas  are 
incorporated  in  command  staff  training  exercises.  The  IB  links  to  logistics  support  will 
provide  training  realism  for  sustainment  of  forces  in  an  operation. 

The  development  of  common  functional  modules  should  be  the  vehicle  for 
including  logistical  support,  C4I  interfaces  in  ADS  federations  for  training  and  analysis 
applications. 


Issue:  M&S  Representation  Scope  and 
its  Impact  on  ADS 

•  OBSERVATIONS: 

-  ADS  typically  addresses  only  part  of  the  scope  of  the 
domain-of-interest  of  analysts  and  training  communities 

-  Much  of  data  gathering,  information  awareness,  and 
knowledge  domains,  are  assumed  away  into  ‘the  real 
C4I  systems’;  operations  support  Is  under-represented 

-  Implications  exist  for  ADS  design  and  system 
specification 

•  CONCLUSION/FINDINGS: 

-  Deserves  attention  by  ADS  development  and  user 
communities 

•  RECOMMENDATION: 

-  Include  full-domain-of-Interest  into  ADS  development 


ISSUE:  M&S  Representation  Scope  and  its  Impact  on  ADS  -  Slide  8 

The  use  of  ADS  usually  addresses  only  a  part  of  the  full  scope  of  the  domain  of 
interest  for  both  the  analysis  and  training  communities.  In  the  JSIMS  concept,  the 
domain  of  interest  of  an  individual  training  event  may  be  played  at  a  greater  level  of 
resolution  or  fidelity  with  distributed  (or  resident)  modules  supporting  the  other 
functional  areas  in  a  more  gross  level  of  detail  or  resolution.  It  was  felt  that  much  of  the 
information  related  data  gathering,  information  awareness  and  warfare/knowledge 
domains  are  assumed  away  as  being  covered  by  the  C4I  systems.  The  C4I  systems  must 
have  seamless  interface  with  other  simulations  and  their  capabilities  enhanced  to 
actually  accommodate  all  the  information  issues  discussed  here.  The  area  of  operations 
support  is  felt  to  be  under-represented  in  the  community.  The  support  of  high  level 
command  and  staff  training  functions  is  a  natural  bridge  between  analysis  and  training. 
Many  of  the  current  tools  for  these  staffs  are  in  fact  analysis-type  models.  In  this 
context  there  are  significant  implications  for  ADS  design  and  detailed  specification  for 
the  emerging  systems. 

The  M&S  level  of  representation  and  domain  of  interest  deserves  attention  by  both 
the  developers  of  our  new  systems  and  by  the  analysis  and  training  community  users. 

The  recommendation  is  that  the  development  agents  resolve  the  full  domain  of 
interest  issue  through  ADS  if  appropriate,  but  also  through  the  process  of  logical  threads 
pulling  the  information  needed  at  the  appropriate  level  of  resolution  to  fuel  the  main  user 
interface/training  event.  The  user  should  have  a  low  sense  of  awareness  that  he  or  she  is 
pulling  from  another  model  or  system. 


Issue:  Potential  Misuse  of  Training  M&S 

Results 


•  OBSERVATIONS: 

-  Training  exercises  offer  a  potential  source  of  data 
for  analytical  efforts 

-  Results  mli  be  employed  in  collateral  analysis 

•  CONCLUSION/FINDINGS: 

-  Need  standards  and  practices  for  use  of  training 
data  for  meaningful  collateral  analysis  and  to 
minimize  misappropriation  of  training  data 

«  RECOMMENDATION: 

-  Address  the  general  issue  of  application  of 
findings  from  ADS  in  domains  other  than  for  their 
originally  intended  use 
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Training  exercises  offer  a  rich  potential  source  of  data  for  analysis  efforts.  Given 
the  availability  of  data  (even  if  only  anecdotal)  from  training  events/exercises,  there  will 
be  use  of  that  data  for  collateral  analysis.  The  problem  that  is  recognized  here  revolves 
around  the  potential  misuse  of  data  from  training  for  analysis  of  resources,  weapons 
system  procurement  decisions  and  so  forth.  This  condition  will  potentially  become  more 
prevalent  with  the  proliferation  of  M&S  applications  for  training. 

It  was  concluded  that  there  exists  a  need  for  standards  and  practices  for  the  use  of 
training  data  for  meaningful  collateral  analysis.  There  should  be  a  recognition  in  both 
the  training  and  analysis  communities  that  the  data  from  training  must  be  taken  in  the 
proper  context  and  used  wisely  to  feed  other  analysis  models  and  studies  for  exogenous 
purposes.  We  need  to  find  ways  to  minimize  the  risk  of  misappropriation  of  training 
data  for  collateral  analysis. 

The  recommendation  addresses  the  larger  issue  of  ADS  and  the  use  of  data  from 
large,  distributed  simulation  federations.  In  general,  the  application  of  data  and  findings 
from  ADS  should  be  controlled  and  used  wisely  for  collateral  analysis  in  domains  other 
than  that  for  their  original  intended  use. 


Issue:  Data  Management 


•  OBSERVATIONS: 

—  ADS  implies  multiple,  distributed  databases 

—  Configuration  Management  (CM)  and  VV&C  are  more 
critical 

-  Historically,  data  management  is  an  analytic  after¬ 
thought 

•  CONCLUSION/FINDINGS: 

-  More  resources  are  needed  for  data  management 

-  Cost  of  multiple,  heterogeneous  exercise-initialization 
databases  can  be  an  inhibitor  to  ADS  use 

•  RECOMMENDATION: 

-  Develop  tools  and  common  data  interface  standards 
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It  was  recognized  that  a  subject  of  the  analysis  and  M&S  comnnnity  is  very  concerned 
with  the  data,  data  management/^pplications  and  certificaticai  issues.  However,  we  feel  strongly 
that  the  tcpic  needed  discussion  in  the  context  of  our  group,  while  recognizing  that  there  is  a  need 
fw  the  development  of  tools  and  standards  in  accordance  with  current  guidehnes  and  regulations. 
It  was  observed  that  ADS  implies  imtiple  distributed  databases.  Ihere  is  wrak  being  done  by 
DMSO  and  their  contractors  in  a  program  to  develop  “Ccaiceptual  Models  of  the  Mssion  Space 
(CMNC).”  This  program  responds  to  the  DMSO  Master  Han  which  advocates  a  ccanmcm 
technical  ftanewotk  thiou^  the  use  of  HLA/CMMS/Data  Standards.  The  CMMS  provides  the 
common  logical  framewoik  for  knowledge  acquisition  and  a  standard  fomiat  for  expressicxi.  It 
will  provide  validated,  relevant  actions  and  interactions  organized  by  specific  task  and 
entity/organization.  The  CMMS  will  be  a  hi^  level,  hiaarchical  representation  of  the 
inteiacticHis  and  actions  of  combat  entitites  in  each  mission  space.  This  will  potentially  become  a 
standard  fcxmat  for  expression  that  will  provide  a  sirndation-independent  source  (rf  infcamaticMi 
c^turing  in  the  data,  tasks,  aitities  and  their  interactions.  Hstorically,  the  area  of  data 
managPinynt  and  sometimes  even  life  cycle  configuration  management  are  analytic  after 
thou^ts. 

Our  conclusion  for  data  is  tiiat  we  need  to  be  investing  more  resources  in  the  data 
management,  configuration  managemrait  and  Verification,  Validation  and  Certification  (VV&C) 
areas  of  ADS  development  fOT  training  ^plications.  Afta:  the  above  discussion,  it  was  also 
recognized  that  on  the  downside  the  cost  of  multiple,  heterogeneous  exadse-initialization 
databases/systems/models  can  be  an  inhibitar  for  ADS  use. 

The  recoramendatian  is  that  the  corrmmity  give  hi^  piority  to  woik  on  tools  and 
common  data  interface  standards.  We  feel  that  the  CMNB  p-ogram  is  a  big  step  in  the  ri^t 
direction. 


Issue:  Analysis/Training  Community 
Interactions 


•  OBSERVATIONS: 

-  Roles  and  missions  of  the  analysis  community  in 
support  of  training  are  not  defined 

-  There  is  no  vision  of  analysis/training  coordination  to 
guide  analysts 

-  Requirements  of  analysis  to  support  training  are 
inexplicit 

•  CONCLUSION/FINDINGS: 

-  Information  exchange  has  been  too  limited 

-  Analysis  community  can  and  will  help  but  terms  needed 

•  RECOMMENDATION: 

-  Improve  communications  between  training  and  analysis 
community 

-  Develop  future  vision,  goals  and  objectives _ 
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Consistent  with  our  earlier  discussion  as  to  the  two  comrmnities  being  “planets  ^jait,”  it 
was  recogni2ed  early  on  that  the  traditional  roles  and  missions  of  the  analysis  community  in 
support  of  training  are  not/have  not  been  well  defined.  There  is  not  currently  an  existing  irodel 
or  vision  of  analysis  in  support  of  training  to  serve  as  guide  for  analysts  and  trainCTS.  We  are 
missions  explicit  requirements  fi’om  the  training  community,  for  analysis  to  siqjport  training.  Part 
of  this  problem  was  discussed  earlier  in  the  MQE  and  the  functional  representation  slides. 

h  is  believed  that  in  the  past  information  exchange  bdween  the  two  communities  has 
been  too  limited.  Another  observation  is  that  the  analysis  community  can  and  will  be  able  to  help 
with  the  thorny  problems  facing  the  training  community,  but  the  requirements  and  an  invitation  to 
play  are  needed. 

The  recommendation  developed  here  to  improve  communications  betweai  the  two 
communities  resulted  in  a  c^)st<aie  recommendaticm  discussed  later  which  will  fcxmalize 
involvement  by  establishing  a  MORS  WG  at  the  annual  Syrrprsiumto  address  die  topic  of 
“Analysis  St^xsrt  for  Training.”  As  vision,  goals  and  objectives  are  developed,  the  communities 
nust  work  together  to  idaitify  areas  of  expertise  and  responsibility  to  share  development  and 
operati(»r  of  simnlatinns,  functional  modules,  common  object  and  data  models  and  the  supporting 
infiastructure. 


Issue:  Multi-Level  Security 


•  OBSERVATIONS: 

-  Still  problematic 

-  Multi-service  coordination,  intelligence  data 
management  cause  difficulties 

-  ADS  network  distribution  exacerbates  problem 

-  Most  operations  default  to  “system-high” 

•  CONCLUSION/FINDINGS: 

-  Much  broader  problem  than  ADS 

•  RECOMMENDATION: 

-  Push  for  resolution  within  DoD  community 
standards  and  practices 
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Tliis  issue  has  been  with  us  fca:  many  years,  but  we  are  only  now  beginning  to 
achieve  the  technological  breakthroughs  that  will  permit  wide  spread,  logical  solutions. 
The  area  is  believed  to  be  STELL  problematic  and  likely  to  cause  more  problems  in  the 
areas  of  raulti-saMce  coordination/cooperation,  intelligence  data  and  information 
warfare  as  we  move  to  more  distributed  environments.  The  ADS  networic  serves  to 
exacerbate  the  probleiiL  In  order  to  manage  the  risk  of  unauthorized  entry  or  disclosure, 
most  operations  will  default  to  a  “system-hi^”  mode,  which  may  hamper  both  training 
and  analysis  ^iplicaticBis  (not  to  mention  the  distribution-over-networks  problems). 

It  was  concluded  by  the  group  that  this  is  a  far  wider  and  deeper  problem  than  just 
the  ADS  portion.  And  although  we  may  not  be  able  to  resolve  the  problem  in  this  forum, 
we  should  recognize  it  as  a  potential  show  stopper  in  the  future  and  wcsk  toward  mutual 
steps  for  resolutioa 

The  training  and  analysis  communities  should  push  for  resolution  with  the  broader 
DoD  community  and  establish  standards,  practices  and  inccxporate  leading  edge 
technologies  to  solve  this  long  standing  and  iruxittant  problem. 


Issue:  Logical  Consistency 


•  OBSERVATIONS: 

-  HLA  supports  technical  means  to  implement  logical 
interactions  among  entities  in  ADS 

-  Logical  unity  of  ADS  ensembles  must  be  designed-in 

-  Critical  to  establishing  a  credible  federation  ensemble 

-  Critical  for  enabling  valid  analyses 

•  CONCLUSION/FINDINGS: 

-  Critical  to  ADS  for  both  training  and  analysis  use 

•  RECOMMENDATIONS: 

-  Establish  suitable  recommended  practices  for  logical 
consistency  of  federation  ensembles 

-  Explore  aggregation/disaggregation  issues 


ISSUE:  Logical  Consistency  -  Slide  13 

A  great  deal  of  discussion  centered  on  the  issue  of  the  representation  of  logical 
interaction  between  model,  modules  or  functional  data  within  and  between  federations. 

It  was  observed  from  Dr.  Dahmann’s  briefing  that  DMSO  developed  HLA  supports  for 
the  technical  aspects  of  this  problem  and  therefore  ought  to  provide  a  technical  means  to 
implement  logical  interactions  among  entities  in  an  ADS  environment.  This  issue  also 
crosses  over  into  the  aggregation/disaggregation  issues  and  the  desire  for  seamless, 
transparent  interface  between  various  entities.  As  discussed  here,  logical  consistency  is 
a  technical  requirement  placed  across  an  ADS  federation  ensemble  for  it  to  be  valid  and 
useful  for  analysis  and  possible  for  training.  It  is  an  extension  of  the  concept  of  a  “fair 
fight”  that  has  become  a  major  concern  within  the  DIS  community.  In  the  DIS  context, 
and  largely  due  to  the  heavy  emphasis  on  visual  representations  in  that  environment,  the 
fair  fight  problem  is  generally  characterized  as  the  need  to  ensure  that  any  interacting 
protagonists  have  the  same  perception  of  reality.  There  are  a  number  of  considerations 
in  the  interface  that  can  cause  problems.  One  is  the  level  of  resolution  and  fidelity  of  the 
representational  entities  and  how  they  interact  within  their  separate  simulation 
environments.  Others  are  time,  terrain,  sensors,  weapons  characteristics,  C3,  etc. 

Logistical  consistency  is  a  more  robust,  macro  level  requirement.  It  means  that 
when  “looking  across”  at  the  ADS  model  ensemble,  the  participants  in  any  potential 
interaction  are  —  or  can  be  made  to  be  —  behaviorally  and  representationally 
consistent.  Representational  consistency  is  more  that  just  the  fair  fight  concept 
described  above.  It  extends  to  the  aggregation/disaggregation  type  problem  as  well. 
Another  concern  is  behavioral  consistency,  which  may  be  even  more  difficult,  largely 
because  it  is  a  far  less  visible,  more  subtle  problem.  The  cure  for  the  behavioral 
consistency  problem  across  an  ensemble  can  be  achieved  either  by  requiring  potentially 
interacting  models  to  be  behaviorally  homogeneous  or  by  making  them  behaviorally 
commensurate.  The  first  of  those  is  too  limiting,  and  the  second  is  presently  too  hard. 
First,  we  need  to  develop  the  standards  that  will  define  the  required  conditions,  and  then 
we  can  begin  the  research  to  meet  those  standards.  But  if  we  are  to  succeed  in  building 
distributed  simulations  for  either/or  both  training  and  analysis,  we  must  address  this 
issue.  It  is  absolutely  essential  to  establishing  a  credible  federation  ensemble  and 
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First,  we  need  to  develop  the  standards  that  will  define  the  required  conditions,  and  then  we  can 
begin  the  research  to  meet  those  standards.  But  if  we  are  to  succeed  in  building  distributed 
simulations  for  either/or  both  training  and  analysis,  we  must  address  this  issue.  It  is  absolutely 
essential  to  establishing  a  credible  federation  ensemble  and  enabling  valid  applications  and 
analyses.The  WG  recommends  the  establishment  of  suitable  practices  to  insure  logical 
consistency  of  federation  ensembles  and  further  explore  the  complexities  of  aggregation/ 
disaggregation. 


Issue:  Training  Exercise  Planning 
Methodology 

•  OBSERVATIONS: 

-  Lack  of  common  exercise-definition  data  and  standards 

-  Need  rigorous  methodoiogy  and  effective  tools  to 
support  exercise  pianning 

-  Exercise  pianning  must  ‘buiid-in’  instrumentation 
sufficient  for  evaluation  of  objectives 

•  CONCLUSION/FiNDINGS: 

-  Pianning  support  is  key  to  exercise  economy 

-  A  (meta-)modei  of  the  exercise  may  be  worthwhile 

-  ADS  technologies  (DBMS,  visualization,  etc..)  may  apply 

•  RECOMMENDATION: 

-  Develop  tools  to  assist  in  exercise  planning  and 
methodology 
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The  concept  of  model-exercise-model  may  help  in  addressing  issues  for  a  specific 
event  and  provide  knowledge  in  designing  a  new  event.  It  was  observed  by  the  group  that 
the  communities  are  absent  a  common  exercise  definition  for  data  and  standards. 
Simulations  should  be  able  to  use  real  world  data  and  have  some  easy  input  for  analysis  of 
the  planning  phase  of  exercises.  It  is  also  necessary  to  establish  rigorous  methodologies 
and  effective  tools  to  support  exercise  planning.  For  instance,  the  user  needs  automated 
development  of  data  bases  and  scenarios.  An  automated,  logical  framework  based  on  the 
mission  essential  task  list  would  be  useful  for  planners,  operators  and  analysts  alike.  The 
CMMS  discussed  earlier  has  some  of  these  attributes,  but  is  intended  for  use  by  the 
model/simulation  developer  —  not  the  end  user.  The  planning  for  exercises  must 
accommodate  the  necessary  instrumentation  sufficient  for  evaluation  of  training  objectives. 

Planning  support  is  also  key  to  economy/efficiency.  Given  the  earlier  discussion  as  to 
the  difference  in  cycle  time  for  tactical  versus  strategic  levels  of  command  staff 
interactions,  a  meta-model  of  tactical  training  could  be  created  by  collecting  the  relevant 
data  produced  by  the  many  lower/tactical  level  training  events.  This  meta-model  could  be 
used  to  feed  the  time  compression  necessary  to  conduct  a  JTF  training  event/exercise  over  a 
period  of  days  instead  of  months  which  would  be  required  for  the  CINC  or  JTF 
Commander’s  staff  in  a  real  time  environment.  It  was  recognized  that  ADS  technologies 
for  data  and  graphical  representation  may  apply,  but  platform  level  of  resolution  seemed 
out  of  place  in  strategic  level  command/staff  exercises. 

We  recommend  that  the  communities  work  together  to  develop  analysis  and  planning 
tools  (and  methodologies)  to  assist  in  the  planning  of  exercises.  Concentration  should  be 
first  on  the  logical,  automated  framework  described  above  that  will  be  used  by  planners 
and  operators  to  help  do  their  jobs  in  the  relatively  short  term. 


Issue:  Analysis  Support  for  AAR 


•  OBSERVATIONS: 

-  Requirement  exists  to  identify  and  analyze  causal 
relationships  and  unusual  events  in  exercises 

-  Need  automated  data  reduction  tools,  consistent  with 
planning  methodology 

•  CONCLUSION/FINDINGS: 

-  AAR  data  should  be  formulated  for  use  by  both 
training  and  analysis  communities 

•  RECOMMENDATIONS: 

-  Develop  AAR  system  to  capture  pre-planned 
automated  data  products 

-  Analyst  and  development  community  coordinate  post¬ 
processing  tool  prototypes 
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AAR  is  an  area  in  which  analysis  can  interact  readily  with  the  training  community. 

The  requirement  exists  to  identify  and  analyze  causal  relationships  and  unusual  events  that 
occur  in  training  exercises.  The  community  needs  automated  data  reduction  tools, 
consistent  with  the  planning  methodology.  The  AAR  can  be  a  logical  extension  of  the 
planning  cycle,  where  the  same  logical  framework  that  plans  and  controls  an  exercise  may 
be  used  to  do  a  rapid  turn  around,  post-mortem  analysis  of  the  training  event. 

The  exercise  data  should  be  formulated  for  use  by  both  the  training  and  analysis 
communities.  Up  front  planning  should  include  the  details  of  data  collection  for  required 
information  to  conduct  analysis  and  the  AAR. 

The  group  recommends  that  the  training  community  work  with  the  analysis  community 
to  develop  a  comprehensive  planning/operating  /AAR  system  to  capture  pre-planned 
automated  data  products.  The  analysis  and  development  communities  should  coordinate 
post  processing  tool  prototypes.  The  ADS  technologies  can  be  useful  in  pulling  together 
the  supporting  data  and  information  for  all  phases  of  the  system. 


Issue:  Rapid  Scenario  Generation  (Piay- 

Revision-Piay) 

•  OBSERVATIONS: 

-  Rapid  scenario  generation  and  efficient  play-revision- 
play  cycles  apply  to  COA  and  operational  mission¬ 
planning  roles 

—  Data  are  critical  components.  Download  of  real  time 
intelligence  data  should  be  available 

•  CONCLUSiON/FINDINGS: 

-  Nothing  in  ADS  (HLA)  contributes  to  solving  this 
problem  —  C4I  technologies  and  practices  will 

•  RECOMMENDATIONS: 

—  Provide  general  support  for  rapid  scenario  generation 
outside  ADS 
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The  Rapid  Scenario  Generation  as  discussed  by  our  WG  relates  to  the  play-revision- 
play  cycles  as  they  apply  to  Courses  of  Action  (COA)  and  operational  mission  planning 
roles.  The  issue  here  is  whether  ADS,  in  the  generic  sense,  enhances,  degrades  or  has 
no  impact  on  the  ability  to  rapidly  generate,  play,  revise  and  replay  scenarios.  This  issue 
arose  while  considering  the  capabilities  and  limitations  of  ADS  in  “Analysis  associated 
with  determining  the  effectiveness  of  ...alternate  courses  of  action  during  operational 
planning.”  In  operational  planning,  unlike  deliberate  planning  activities,  analysts  need 
the  ability  to  rapidly  set  up  and  test  and  evaluate  a  multitude  of  COAs  in  a  very  short 
period  of  time.  The  turnaround  times  of  the  operational  planning  cycle  are  measured  in 
minutes  or  hours  (a  very  few  hours)  and  not  in  days  or  weeks.  As  one  aspect  of  the 
problem,  operational  planning  MUST  have  access  to  the  most  current  intelligence  data. 
Today,  that  means  real  time  data.  A  simulation  supporting  this  type  of  planning  process 
must  be  able  to  keep  up  with  the  compressed  data  flow  —  which  will  drive  analytic 
turnaround  to  near  real  time  responses. 

The  definition  of  ADS  becomes  important  in  the  above  context.  As  indicated 
earlier,  WG4  did  not  try  to  redefine  ADS,  but  was  aware  of  the  definition  provided  to  all 
ADSA  ‘96  participants  by  Sikora  and  Goose  in  the  June  ‘95  edition  of  PHALANX.  This 
definition  includes  the  geographically  distributed  environment  for  interactive 
simulations  which  may  diminish  the  capability  of  ADS  to  respond  in  the  rapid  scenario 
generation  situation.  Some  of  the  advantages/disadvantages  of  ADS  over  a  stand  alone 
simulation  are: 
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•Time  consumption  impact  of  the  data  management  problem  that  is  inherent 
to  multiple  distributed  databases  (a  particularly  severe  problem  when  the  data 
is  perishable  intelligence  data  with  a  high  potential  for  multi-level  security 
problems, 

•Time  consumption  impact  of  identifying  and  resolving  new  latency  and 
logical  consistency  problems  injected  whenever  simulations  are  added  or 
deleted  from  an  ADS  ensemble, 

•Inherently  greater  functional  failure  potential  of  an  ADS  ensemble  over  a 
stand  alone  simulation 

After  discussion  of  considerations  such  as  these,  the  recommendation  was  that  the  general 
support  for  rapid  scenario  generation  should  be  provided  by  means  other  than  ADS-based 
systems. 


Issue:  Commonality  of  ADS  for 
Analysis  and  Training 

•  OBSERVATIONS: 

-  ADS  M&S  framework  suitable  for  training  and 
analysis 

•  CONCLUSiON/FINDINGS: 

-  Logicai  consistency  is  attainable  but  may  be 
expensive 

-  Applications  drive  the  details 

•  RECOMMENDATIONS: 

-  Training  and  analysis  must  cooperate  on  ADS 
design  and  federation  development  (opportunity) 

-  Must  design  and  employ  federations  for  ‘intended 
use’  (responsibility) 
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It  was  observed  that  the  ADS  M&S  framework  can  be  suitable  for  both  training  and 
analysis  applications.  The  question  is  how  one  achieves  the  logical  consistency  as 
described  earlier  and  at  what  cost.  A  well  defined  M&S  infrastructure  needs  to  be 
developed  that  will  ensure  easy  access  to  M&S  information,  data  and  models.  The 
applicability  and  use  of  M&S  should  in  itself  become  a  matter  of  military  professional 
education  to  ingrain  the  somewhat  standard,  structured  analysis  process  in  the  military 
trainers  and  developers. 

Logical  consistency  is  concluded  to  be  attainable  in  an  ADS  environment,  but  it 
may  be  very  expensive.  The  cost/benefit  determination  discussed  earlier  should  be 
applied.  The  driving  reason  to  move  to  a  common  ADS  environment  should  be  based 
on  the  specific  application  and  the  requirements  for  training,  not  the  technical  feasibility 
of  hooking  the  pieces  together. 

The  first  recommendation  in  this  area  is  that  the  training  and  analysis  communities 
cooperate  and  collaborate  on  ADS  design  and  federation  development.  That  is  our 
opportunity.  The  second  recommendation  is  related.  The  two  conununities  must  design 
and  employ  federations  for  their  “intended  use”  and  with  only  their  intended  use 
forefront  to  avoid  costly  mistakes  leading  to  unsatisfactory  results.  That  is  our 
responsibility. 


Issue:  Commonality  of  VV&A  In  ADS  for 
Analysis  and  Training 

•  OBSERVATIONS: 

-  The  same  VV&A  practice  is  suitable  for  both  training 
and  analysis 

-  Application/use  is  the  discriminator 

-  There  are  aspects  of  ADS  which  require  VV&A  tailoring 

•  CONCLUSION/FINDINGS: 

-  No  essential  difference  in  the  type  of  applicable 
techniques 

-  Security  and  data  management  are  important  and 
related  considerations 

•  RECOMMENDATION: 

-  Analysis  and  training  community  together  participate  in 
determination  of  ADS  VV&A  practices 
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One  of  the  most  important  considerations  with  the  use  of  M&S  for  either  analysis  or 
training  is  the  confidence  of  the  user  and  the  “rightness”  of  the  tool  for  the  specific 
application.  W&A  is  a  complex  area  that  has  troubled  both  the  developers  and  users  of 
M&S  for  years.  There  is  a  lively  (and  long  running)  debate  in  the  analysis  community  as 
to  how  much  VV&A  is  enough,  and  whether  M&S  users  are  being  either  insufficiently 
demanding  and  supportive  of  the  outcome  on  one  extreme,  or  requiring  a  degree  of 
validation  that  is  impossible  on  the  other.  All  serious  players  agree  on  the  need  for 
VV&A,  but  may  differ  considerably  on  the  degree  or  level  of  W&A  activity  that  is 
appropriate  for  a  given  model  and  application.  It  was  observed  by  the  group  that  common 
(established)  W&A  practices  are  suitable  for  both  training  and  analysis  applications.  The 
end  application  or  use  of  the  model/simulation  system  should  be  the  discriminator  as  to 
what  level  of  activity  is  enough,  and  the  aspects  of  ADS  will  definitely  require  W&A 
tailoring  and  activity. 

It  was  concluded  that  there  is  no  essential  difference  in  the  “kind”  of  applicable 
W&A  techniques  when  used  for  analysis  or  training  applications.  The  issues  of  security 
and  data  management  which  were  discussed  on  earlier  slides,  are  felt  to  be  very  important 
related  issues/considerations  for  the  life  cycle  for  the  -W &A  of  models.  A  new  DODI 
5000.XX  has  been  written  and  coordinated  to  lay  out  a  common  framework,  process, 
methodologies,  responsibilities  and  definitions  as  they  are  applied  across  the  DoD. 

It  is  recommended  that  the  analysis  and  training  communities  should  jointly 
participate  in  the  determination  of  the  appropriate  and  standard  W&A  practices  for  ADS 
systems  and  their  applications. 


Issue:  Relationships  of  Analysis  and 
Training  vis-a-vis  ADS 

•  OBSERVATIONS: 

-  Clarifying  the  interesting  and  useful  relationships 
between  analysis  and  training  is  unexpectedly  difficult 

-  This  taxonomic  meta-analysis  was  worked  In  parallel 
with  other  issue  analysis 

«  CONCLUSION/FINDINGS: 

-  Producing  such  a  correlation  or  field  map  ought  to  be 
useful  in  providing  structure  for  foliow-on  deliberation 
or  action 

«  RECOMMENDATION: 

-  Develop  and  use  an  analyst-training  relationship 
representation 
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As  indicated  in  the  introduction,  the  analysis  and  training  communities  are  planets 
apart  when  it  comes  to  working  together  to  provide  timely  operational  analysis  to  training 
exercises.  This  observation  is  independent  of  the  thematic  content  of  ADS  for  Analysis, 
but  is  applicable  across  the  full  spectrum  of  analysis  in  support  of  training.  First,  the 
group  observes  that  it  will  be  necessary  to  clarify  the  interesting  and  useful  relationships 
between  training  and  analysis.  This  first  step,  we  find  is  unexpectedly  difficult.  There  are 
undoubtedly  reasons  for  this  that  go  back  to  the  early  history  of  Operations  Research  (OR) 
and  the  more  recent  phenomenon  of  the  linking  of  the  stand  alone  simulation  systems  for 
training  applications.  The  insights  into  this  problem  were  manifest  from  the  Group  which 
consisted  of  both  analysts  and  operational/training  experts.  Several  of  the  issues  chosen 
for  development  by  the  group  lead  to  recommendations  that  the  two  communities 
cooperate  in  ways  that  are  apparently  not  happening.  The  single  most  meaningful 
contribution  of  this  group  may  be  the  first  steps  in  forming  a  bridge  between  the  two 
worlds. 

In  true  OR  fashion,  the  analyst  must  incorporate  subject  matter  experts  and  get  to 
know  the  users  of  the  resultant  products  of  analysis.  That  means  in  this  case,  analysts 
spending  more  time  getting  to  know  the  systems,  processes,  policies  and  procedures  of  the 
training  community.  The  resultant  correlation  was  felt  to  provide  a  useful  basis  and 
structure  for  follow-on  deliberation  and  action  between  the  communities. 

The  recommendation  resulting  from  this  discussion  was  to  develop  and  use  an 
analyst-training  relationship  representation  that  will  form  the  foundation  for  analysis 
support  for  training.  An  example  of  this  representation  in  matrix  format  is  found  on  the 
next  slide. 


ANALYSIS-TRAINING  Relationships, 
(Continued) 


ANALYSIS 

DOMAIN 


PROCESS: 

1 .  Define  dimensions 

2.  Review  ceiis 

3.  Compile  synopsis 
and  net  assessment 


KINDS  OF  RELATIONSHIPS: 

..  A  is  used  for  x  on  behalf  of  B 
..  A  and  B  both  depend  on  x 
..  A  and  B  are  discriminated  by  x 


FOR  EACH  CELL,  CONSIDER: 

What  form  of  relationship  exists? 

Are  there  issues  of  general  concern? 
Are  there  issues  of  concern  WRT  ADS? 
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The  matrix  shown  here  is  a  conceptual  representation  of  how  an  analyst 
could,  with  some  rigor,  produce  a  mapping  of  the  analysis-training  relationships. 
It  is  stressed  that  this  is  only  one  quick  attempt  at  producing  a  logical  framework 
for  this  issue. 


Recommendation 


•  RECOMMENDATION: 

-  Establish  MORS  WG  Analysis  Support  for  Training 
commencing  at  the  65th  MORSS 

«  MiSSION: 

-  To  facilitate  coordination  between  the  analysis  and 
training  communities 

•  INITIAL  TASKING:  (ORGANIZE  AT  64TH  MORSS) 

-  Address  methods,  techniques  and  tools  for  the 
analysis  community  to  support  training: 

»  Exercise  support  planning 
»  AAR 

»  Effectiveness  of  training 
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This  chart  is  titled  simply  “Recommendation,”  because  it  is  the  result  of  many  of  the  issues 
and  recommendations  which  have  gone  before.  The  current  structure  of  our  annual  MORS 
Symposium  (MORSS)  does  not  devote  a  single  working  group  strictly  to  training.  Training  is 
discussed  at  the  MORSS,  but  only  peripherally  in  several  of  the  existing  groups.  Currently,  no 
one  working  group  focuses  on  analysis  for  training  support  as  part  of  its  charter.  Discussions 
which  took  place  in  ADSA  ‘96  between  the  operators,  JSIMS  JPO,  JWFC  Technology  Division 
and  analysts  presented  a  number  of  issues  found  in  our  earlier  charts  that  indicate  a  real  need  for 
analytic  support.  These  issues  addressed  AAR,  exercise  planning  and  support  and  training 
evaluations  —  all  areas  that  have  been  studied  over  the  years.  With  today’s  focus  on  joint 
training  and  command/staff  interaction  during  JTF  operations,  the  topic  bears  increased 
attention  by  both  communities. 

The  MORS  leadership  will  bring  before  the  Board  of  Directors  (BoD)  at  the  June  1996 
meeting  a  proposal  to  establish  a  WG  for  our  MORSS  devoted  to  “Analysis  Support  for 
Training”  to  commence  during  the  65th  MORSS  in  Quantico,  VA  during  June  of  1997.  The 
mission  of  this  group  will  be  to  facilitate  coordination  between  the  analysis  and  training 
communities.  Given  approval  by  the  MORS  Board,  an  organizing  meeting  will  be  held  during 
the  64th  MORSS  this  coming  June  at  Fort  Leavenworth,  KS.  More  details  will  be  provided  (as 
feedback  and  invitations  to  participate)  to  interested  training  organizations/individuals  after  our 
June  organizational  meeting. 


Advanced  Distributed  Simulation  for 
Analysis 

Working  Group  5 
(Analysis  and  Requirements) 


MORS  Workshop 
Williamsburg,  Virginia 
30  January  - 1  February  1996 
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WG  5,  responsible  for  Analysis  and  Requirements,  was  chaired  by  Dr.  Cyrus  Staniec  of 
OSD(PA&E),  and  assisted  by 
Co-Chairs:  CoL  Paul  Hanover,  USMC 

Ms.  Marchelle  Stahl,  IDA 

Mr.  Steve  Stamer,  SAIC  (DMSO  Technical  Support) 

The  Group  had  about  twenty  participants,  representing  a  wide  experience  base  — 
primarily  analysts,  salted  with  engineers  and  M&S  technologists.  All  of  the  Services,  OSD  and 
BMDO  were  represented.  Major  DoD  analytic  organizations,  FFRDCs,  academia  and  defense 
industry  were  present.  Analytic  experience  was  heavy  in  the  areas  of  battlespace  effectiveness, 
acquisition  and  training/rehearsal.  This  Group  clearly  had  the  depth  and  breadth  to  address  the 
analytic  issues  at  hand,  (However,  one  interesting  post-Workshop  self-assessment  question  at 
the  end  of  the  conference  was  whether  we  had  enough  involvement  of  the  customer  —  the 
consumers  of  our  proposed  analysis...) 
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Comparison:  DIS  Mini-Symposimn  to  ADS  Workshop 


WGI:  Military  Analysis 
WG5:  Force  Development 
WG6:  Combat  Development 


WG8:  Requirements  Development 
WG9:  Simulation  Prototyping  (Acq’n) 
WG2:  Test  &  Evaluation 
WG3:  Operations  Planning/ReheafsaT 
WG7:  Training  &  Readiness 
WG4:  Logisdcs/Mobilization/Sustain 
WGIO:  Cost/Benefit/RiskofDIS 


WGI:  Battlefield  Effectiveness 

WG2:  Materiel  Systems  &  Acquisition 

WG3:  Test  &  Evaluation 

WG4:  Tiaining/Rehearsal/COA  Analysis 

WG5:  Analysis  &  Requirements 

WG6:  Understanding  Behavior  &  Performance 

WG7:  Synthesis 


•T6RDes(a^>tiontpr  WG$:  </li»^  if  :  '  ^ 
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The  ADSA  Workshop  Terms  of  Reference  (TOR)  chartered  WG  5  to  examine  analysis 
and  requirements  issues  applied  to  ADS,  “including  processes  for  assessing  and  improving 
ADS-based  analytic  tools.”  The  WG  took  this  definition  as  a  cross-cutting  charter  to  look  at  all 
possible  opportunities  for  analysis  applications.  This  point  of  view  is  illustrated  in  this  slide 
mapping  working  groups  from  the  first  MORS  DIS  Mini-Symposium  into  the  ADSA  working 
groups.  Since  the  major  functional  categories  from  DIS  mapped  well  into  the  the  first  four 
ADSA  working  groups,  we  felt  free  to  look  for  an  overarching  view. 


MORS  ADSA  Workshop 

WG  5:  Functional  Area  Examples 


•  Three  functional  area  examples  were  briefed  to  provide  a  concrete  basis  for  analytic 
applications  and  issues: 

-  “Applying  Synthetic  Environments  to  Operational  Training  -  A  Perspective  from 
Kernel  Blitz  95”  by  Mr.  Tom  Neuberger  from  Center  for  Naval  Analyses 

-  “Simulation  Based  Design”  by  Mr.  Mike  Roberts  from  the  Navy  Acquisition  Reform 
Office 

-  “Anti-Armor  Advanced  Technology  Demonstration  (A2  ATD)”  by  Mr.  Wilbert  J. 
Brooks  from  US  Army  Materiel  Systems  Analysis  Activity 


m  OSD/PA&E 
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To  ensure  a  broad,  even  understanding  of  the  demands  and  opportunities  for  analysis  across 
function  areas,  the  WG  received  briefings  on  three  current  ADS-based  programs  spanning  the  four 
major  functional  areas: 

1.  “Applying  Synthetic  Environments  to  Operational  Training  -  A  Perspective  from  Kernel 
Blitz  95”  by  Mr.  Tom  Neuberger  from  Center  for  Naval  Analyses.  This  briefing  described  a  major 
ADS  exercise  conducted  by  the  Navy  in  the  Spring  of  1995.  Models  and  infrastructure  used  to 
support  KemelBIitz  95  were  described  and  several  issues  concerning  analysis  were  highlighted.  Mr. 
Neuberger  indicated  that  ADS  is  more  applicable  to  certain  military  operations  than  to  others.  Live, 

virtual  and  constructive  linkage  is  not  viable  for  all  players  and  purposes  (e.g . .  Navy  C2  structure, 

submarine  operations  and  carrier  flight  deck  personnel).  There  is  a  lack  of  analysis  tools  to  support 
ADS  analysis  and  no  standard  data  recording  formats.  Lastly,  one  should  be  cautious  about  the 
outputs  of  an  analysis  conducted  using  ADS  because  sophisticated  displays  often  cover  simulation 
flaws  and  man-in- the-loop  doesn’t  necessarily  translate  into  credibility  of  output. 

2.  “Simulation  Based  Design”  by  Mr.  Mike  Roberts  from  the  Navy  Acquisition  Reform 
Office  described  ARPA  efforts  to  use  ADS  to  support  the  design  of  new  weapon  systems.  ADS  was 
lauded  as  a  technology  with  unlimited  potential  to  provide  insight  into  the  collaborative  efforts  of 
designers,  developers  and  program  managers,  alike. 

3.  “Anti-Armor  Advanced  Technology  Demonstration  (A2  ATD)”  by  Mr.  Wilbert  J  Brooks 
from  US  Army  Materiel  Systems  Analysis  Activity  provided  an  insightful  overview  of  the  process 
developed  to  ensure  a  credible  ADS  experiment  was  conducted,  including  a  summary  of  the  results 
and  insight  into  lessons  learned.  The  first  integrated  DIS  capability  was  demonstrated  by  A2  ATD. 
DIS  analytical  tools  were  developed  and  demonstrated,  W&A  approaches  were  designed  and 
conducted  on  several  systems,  and,  most  importantly,  a  process  for  conducting  a  credible  DIS 
evaluation  was  developed  and  demonstrated. 
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Group  5  Working  Definition  of  ADS 


•  Simulations  interoperating 
through  established  protocols, 
in  a  concurrent  mode  and 
possessing  the  following 
qualities 


4 


-  Agreed  standards  of  common 
object  behavior 

-  Shared  simulation  services 


•  Attributes: 

^  -  Interoperable 


u 


-  Iiiiuiiieiive  (iiiau-iu-llie-loop) 

-  Distributed 

-  Advanced _ _ 

-  Level  ol  Resolution 
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WG  five  developed  a  working  definition  of  ADS  starting  from  the  outline  above.  This 
definition  is  purposely  quite  broad  and  brings  one  of  the  Group’s  requirements  for  analysis  to 
the  discussion. 

The  Group  accepted  the  fundamentals  of  distribution  —  HLA,  RTI,  etc.  —  quite 
straightforwardly,  as  well  as  the  concept  of  interoperability.  However,  man-in-the-loop  is  not 
required  by  this  definition  {though  permitted)  and  resolution  need  not  be  entity-level  to  qualify. 
The  following  examples  would  be  characterized  as  ADS  applications  according  to  this 
definition.  They  illustrate  the  breadth  of  ADS  and  are  not  meant  to  be  limiting. 

•  Man-in-the-loc^,  both  live  and  virtual  simulations. 

•  Interoperable  constructive  simulations,  in  which  the  level  of  entity 
aggregation  may  be  high  or  entities  may  be  integrated  at  the  platform/system 
level. 

•  A  distributed  collaborative  environment  to  support  interaction  among  several 
communities  that  may  be  involved  in  planning  and  conducting  an  exercise. 

This  environment  might  be  supported  by  GroupWare  and  a  VTC  capability. 

(Postscript:  Dr.  Paul  Davis  made  the  point  during  review  of  this  report  that  analysts  could 
relax  this  definition  further  by  removing  the  '^protocols**  requirement.  Certainly  being  able  to 
distribute  models  through  an  intelligent  interface  not  requiring  a  protocol  would  qualify  as 
advanced,  ”  and  have  significant  attraction  to  analysts.  He  would  use  a  modifier  to  define 
when  a  protocol  is  included,  such  as  Protocol-Mediated  ADS  (PMADS),for  example.) 
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Matrix  Development:  Analytic  Issues/Applications 

Battlespace 

Acquisition 

Training/ 

Test& 

Effectiveness 

Rehearsal/COA 

Evaluation 

•  Force  structure  analysis 

•  Rapid  prototvping/human 

•  Individual/crew  training 

•  Test  design 

-  Force  sizing/ 

factors 

-  Performance/effectiveness 

•  Simulator  VVA 

requirements/capabilities 

•  Requirements 

•  Collective  (unit)  training 

•  Test  “sunrogation” 

-  Force  design/mix 

Determination 

-  Pcrformance/efTectiveness 

•  Model*Test-Model 

>  SupportabOity 

-  Design 

•  Fleet^orce  training 

•  Test  evaluation 

•  Combat  devdopment 

-  ORD 

>  Exercise  design  analysis 

..  Tactks/doctrine  analysis 

•  Doctrine/tactics/BOl 

-  OP  outcome  analysis 

•  Special 

development 

-  Training  effectiveness 

..  C41  Analysis 

•  COEA 

Analysis 

•  Battlefield  awareness 

-  ‘Support”  analysis  (tools) 

•  Information  warfare 

*  Mission  rehearsal 

-  Human  performance 

•  Tactks/doctrine  evaluation 

•  Course  of  action  analysis 

In  preparation  for  a  detailed  review  of  the  various  functional  areas  of  analysis,  this 
table  of  analytic  applications  was  developed.  Not  intended  to  be  comprehensive,  it  provided  a 
basis  for  evaluating  ADS  potential.  Each  application  presents  different  demands  for  the  level 
and  scope  of  the  associated  simulation,  and  therefore  presents  differing  levels  of  opportunity  and 
demand  on  analysis.  The  Group  split  into  two  sub-groups  to  develop  discussion  of  ADS 
applicability,  opportunity  and  pitfalls  for  the  battle  space  effectiveness,  acquisition  and  training 
functional  areas.  T&E  was  not  addressed  explicitly  due  to  lack  of  time. 
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WG  Scope:  “Address  Broad  Analytic  Questions  within  [your]  Domain” 

ref:  Terms  of  Reference  (TOR) 


•  When  should  ADS-based  capabilities  be  used? 

•  Are  ADS-based  capabilities  better  or  worse  for  specific  types  of  analysis? 

•  Does  ADS  allow  us  to  address  any  new,  necessary  analytic  questions? 

•  How  do  ADS-based  capabilities  fit  with  other  types  of  analytic  capabilities  and  approaches? 

•  What  are  the  limitations  of  current  and  future  ADS-based  capabilities? 

•  Are  there  pitfalls  in  analysis  using  ADS-based  capabilities? 

•  Does  the  use  of  ADS-based  capabilities  affect  the  utility  of  analytic  results? 

•  Are  there  useful  ways  to  utilize  the  results  from  ADS-based  analysis,  exercise  and  training  that 
differ  from  conventional  methods? 

•  Do  ADS-based  analysis  capabilities  make  the  VV&A  processes  any  harder,  easier  or  different? 
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The  TOR  provided  the  questions  listed  above  to  help  focus  responses.  WG  five 
reviewed  these  questions  and  incorporated  their  essence  into  its  discussions,  but  deferred  direct 
discussion  of  them  until  after  it  reviewed  selected  functional  areas.  The  flavor  of  WG  five’s 
responses  is  captured  in  the  following  slides.  Specific  responses  to  the  questions  are  provided  at 
the  conclusion  of  this  briefing. 

The  task  for  each  subgroup  was  to  examine  at  least  one  specific  analytic  application  in 
its  assigned  functional  areas  to  assess  potential  value  added  by  using  ADS,  Where  potential 
was  indicated,  the  assignment  was  to  identify  opportunity  and  pitfalls,  then  attempt  to  fathom 
requirements  necessary  to  achieve  the  highest  potential  of  the  ADS  -  analysis  interface.  One 
subgroup  considered  battlespace  effectiveness  and  COA  analysis,  while  the  other  worked  on 
acquisition  and  training/rehearsal.  Upon  review  of  the  integrated  product  of  the  subgroups, 
certain  trends  emerged,  leading  to  a  few  overarching  principles. 
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WG  Scope:  Vision  for  Analysis  using  ADS 


Education  is  tfie  Mortar 
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The  detailed  observations  of  the  subgroup  for  each  functional  area  are  presented  in  the 
following  slides.  However,  the  detailed  observations  led  to  a  set  of  generalized  principles  which 
we  described  as  the  “analytic  vision”  depicted  above. 

The  thesis  of  the  vision  is  that  there  is  potential  for  a  quantum  increase  in  the  value  of 
analysis  based  in  ADS  —  if  we  construct  the  right  capabilities  on  the  right  foundation.  The 
foundation  is  the  connectivity  provided  by  the  physical  communication  network,  the  high  level 
architecture,  run  time  infrastructure  and  common  object  models.  The  pillars  of  progress  are: 
development  of  broad  data  support  and  suitable  MOEs,  improved  scenario  management  tools 
designed  to  assist  in  setting  up  and  executing  analysis,  a  family  of  models  integrated  to  span  the 
spectrum  of  conflict  with  appropriate  levels  of  resolution  and  interfacing  procedures  and  a  set  of 
tools  to  support  analysis  in  this  advanced,  but  more  demanding,  environment. 

The  overarching  goal  of  the  vision  is  an  environment  supporting  distributed  collaborative 
analysis  —  in  which  analysts  are  able  to  interact  from  their  various  distributed  locations  to 
conduct  all  aspects  of  analytic  effort. 

This  includes  the  distributed  execution  of  models,  data  development  and  distribution, 
planning,  sharing  of  analytic  work  and  telecommunications.  The  value  of  these  contributions  is 
described  in  the  ensuing  charts. 
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Working  Group  5:  General  Assessment 


•  ADS  benefits  analysis  in  most  areas 

-  At  some  level,  in  differing  degrees 

-  With  preparation  of  processes,  tools  and  analysts 


-  Under  right  circumstances 
•  Time 


^  Resolution 


•  Resources 

•  Need  for  interaction/collaboration 

•  Need  for  appropriate  detail 


Increasing  resolution,  scope  of  effort,  and  number  of 
participants  (sites)  causes  an  increase  in  complexity  for 
analytic  applications  which  maybe  unwarranted,  even 
confounding! 


- ^ 

Scope 
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This  graphic  depicts  the  “complexity  continuum”  of  ADS-based  analysis.  As  the 
degree  of  resolution  and  the  number  of  distributed  participants  increases,  the  simulation  itself 
becomes  more  complex.  As  the  scope  of  the  simulation  or  analysis  grows,  complexity  increases 
further.  The  utility  of  ADS  for  analysis  depends  on  whether  the  complexity  of  a  particular 
application  is  appropriate  and  affordable  for  the  analytic  need. 

Some  considerations  in  evaluating  when  ADS  adds  value  to  an  analysis  include  the 
time  available,  the  amount  of  interaction  required  and  the  type  and  amount  of  data  available  that 
is  required  for  the  analysis. 

Resolution  ranges  from  small  unit  tank  exercises  that  are  relatively  easily  instrumented 
and  information  captured,  to  the  need  for  theater  level  analysis,  where  the  possible  constructive 
simulation  of  100,000  entities  is  not  desired.  A  force-level  COA  analysis  requiring  very  quick 
turn  around  may  not  be  compatible  with  high  resolution  ADS,  both  due  to  time  constraint  and 
undue  complexity.  On  the  other  hand,  a  single  carefully  scripted  exercise  with  larger  scope  and 
more  detail  may  support  several  analyses  of  lesser  scope.  Human  interaction  provides  for 
credible  insertion  of  key  subjective  factors;  NBC,  C4I,  political  nulitary  factors,  etc. 

Here  again,  we  make  the  point  that  distribution  of  the  analysis  may  be  more  in  terms  of 
the  overall  process,  as  opposed  to  the  simulation  entities  and  engines  themselves.  In  either 
event,  developing  the  aspects  of  ADS  that  support  collaboration  among  the  participants  is  rated 
as  an  important  improvement  of  the  analytic  process. 
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WG  5:  ADS  Requirements  to  support  Force-Level 
Effectiveness  Analysis/Course  of  Action  Analysis 

•  Relevance  of  ADS  to  support  theater-level  analysis 

-  Use  ADS  to  calibrate/revalidate/generate  data 

»  Platform-level  application  undesirable  in  principle  and  practice 

-  Use  ADS  to  distribute  model/data  support 

•  Distributing  process  multiplies  analyst  resources/enables  collaboration 

-  C4I  representation/effect  added,  and  “across  the  board”  enhancement 

-  Warfighter/decision  maker/subject  matter  expert  in  process 

•  Requirements 

-  Prxx:ess  and  support  data  generation 

-  Facilitate  building  integrated  family  of  models  (proactive  prep  for  analysis) 

»  Design  exercise  (experiments)  to  collect  data 
»  Define  data  forms  required 
»  Build  data  collection  and  processing  tools 
~  Develop  tools  for  distributed  collaborative  planning^execution  of  analysis 

-  Solve  scenario  development  and  management  dilemma 

-  Provide  tools  for  rapid  turn  decision  support 

-  Connect  to  all  players  ((jCCS) 
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Considering  the  requirements  typically  found  in  theater  force  analysis,  the  following  issues  in 
applying  ADS  to  analysis  arise: 

1.  Successful  simulation  of  large  numbers  of  high-resolution  entities,  while  required  for  realistic 
training  simulations,  are  not  required  for  theater  level  modeling.  The  interactions  of  multiple  entities 
confound  the  analytical  results.  Aggregate  models  provide  the  best  basis  for  theater  level  analysis. 

2.  Collaboration  (both  vertical  and  horizontal)  provides  perhaps  the  most  value  added  to 
analysis.  It  will  provide  a  capability  to  more  easily  bring  together  analytical  agencies  and  extend  the 
range  of  variables.  The  use  of  ADS  will  provide  an  ability  to  harness  many  resources  in  a  short  period  of 
time  to  support  both  crisis  planning  and  COA  analysis. 

3.  The  ability  to  include  a  hum^i  decision  maker  in  the  analysis  system  will  allow  better 
representation  of  C4I  and  permit  more  detailed  C4I  analysis.  There  is  potential  for  an  across-the-board 
enhancement  in  forces  and  processes. 

4.  The  warfighter  and  the  decision  maker  can  be  involved  in  both  the  ADS  exercise  and  be 
allowed  to  visualize  the  exercise  and  the  results  using  advanced  techniques,  such  as  Virtual  Reality  (VR). 
This  facilitates  both  user  feedback  and  sponsor  acceptance  of  the  results. 

5.  ADS  allows  for  HITL,  insuring  that  difficult  concepts,  such  as  new  doctrinal  concepts  and 
C4I,  are  creditably  and  accurately  represented. 

6.  The  development  of  supporting  tools  is  essential  for  successful  use  of  ADS  in  analysis. 
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Nowhere  else  in  our  deliberations  of  the  relevance  and  merits  of  ADSA  did  an  obvious 
need  emerge  so  clearly  as  in  ADSA  support  to  the  acquisition  process.  Mr.  Mike  Roberts, 
acquisition  reform  leader  within  die  DoN  RD&A  office,  briefed  convincingly  about  a  vision  of 
distributed  information  technologies  in  support  of  acquisition,  and  we  quickly  accepted  him  as 
the  customer  of  our  discussion  results. 

In  its  sin^lest  form,  the  current  acquisition  process  is  stratified  by  DoD  5000  into  diree 
major  stages:  concept  development,  system  development  and  T&E.  Most  Services  precede  this 
with  a  requirements  definition  phase  (often  referred  to  as  “pre-milestone  0”  work).  Insofar  as 
milestone  decisions  are  sequential,  and  demand  repeated  (and  sequential)  cost-benefit  analyses, 
the  whole  process  involves  time-phased,  linear  product  development. 

Mike  Robertas  brief  called  for  a  central  repository  of  concurrently  shared  information 
pertaining  to/describing  a  target  widget,  currently  in  the  acquisition  process.  He  envisions  that 
training  personnel,  analysts,  engineers  and  manufacturers  could  access  appropriate  parts  of  the 
widget’s  data  files  in  order  to  complete  their  respective  pieces  of  the  acquisition  effort. 
Furthermore,  he  says,  changes  driven  by  one  user  community  would  become  immediately 
visible  and  available  to  the  others.  Rather  like  Microsoft’s  OLE,  right? 


Our  WG  developed  Mike’s  vision  from  the  ADSA  standpoint.  This  slide  shows  the 
relationships  and  process  dependencies  we  defined. 

•  The  target  widget  is  described  in  an  object  library  which  contains  a  suite  of  widget 
models,  built  to  the  varying  degrees  of  fidelity  appropriate  to  the  users  in  the 
simulation  confederation. 

•  The  various  widget  data  users  are  grouped  arbitrarily  and  depicted  in  the  clouds. 

•  The  stubby  two-headed  arrows  depicts  the  ADS  network  connectivity. 

•  The  curved  two-headed  arrows  represent  the  distributed,  parallel  collaboration 
effort  we  determined  was  essential  to  the  success  of  this  process  (described  more 
completely  below). 

The  message  of  the  slide  is  threefold.  First,  it  connotes  the  concurrency  of  the  interaction  of 
data  users.  Otherwise  stated,  it  intends  to  substantially  revise,  if  not  fundamentally  redefine,  the 
formerly  sequential  process.  Secondly,  it  denotes  and  supports,  the  vital  involvement  of  the  training 
and  operations  community  in  every  iteration  of  the  product  definition.  And  finally,  it  asserts  that  there 
is  one,  and  only  one,  authoritative  representation  of  the  widget  (although  of  various  levels  of  detail) 
for  use  by  planners,  analysts,  operators,  engineers,  supporters  and  naysayers,  budgeteers,  trainers,  et 
al.  This  vision  would  exploit  ADS  in  every  sense.  If  accepted  and  implemented,  it  would  help 
analysts  in  every  functional  area  substantially  reduce  time  in  virtually  every  phase  of  product 
development,  manufacture  and  introduction  to  the  user.  However,  making  this  happen  obviously 
entails  some  challenges. 
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Relevance  of  ADS:  Issues  in  Integrating  the  Acquisition  Process 


•  W&A  policy  and  practices 

•  Simulation  architecture  specifications 

»  Shared  attributes  and  methods 
»  Shared  contextual  information 

•  Common  shared  environment 

-  Across  multiple  physical  domains  and  spectra 

•  HLA's  RTI  Specification  Must  Support 

-  Variable  time  control  (R/T,  FTRT,  start/stop/restart) 

-  Analytic  setup  and  data  capture 

-  Setup  and  runtime  data  analysis  and  capture  tools 

•  Requirements  to  support  people: 

-  Collaborative  tools  (parallel  network:  GroupWare,  VTC) 

-  Education/training  of  participants 

•  *  Management  process  change  required;  cultural  acceptance  may  be  a  problem 
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This  slide  presents  issues  which  we  considered  most  daunting  in  making  this 
vision  a  reality.  As  might  be  expected,  the  really  hot-button  item  of  W&A  provoked  an  hour  of 
discussion.  Full  appreciation  of  the  complexity  of  the  V&V  process  was  enjoyed  by  all.  In  fact 
we  concluded  with  the  finding  that  our  attitudes  and  expectations  with  regard  to  W&A  would 
need  to  change,  because  the  feasibility  of  V&Ving  n  interacting  simulations  was  actually  an  nn- 
1  problem,  assuming  static  simulation  relationships.  If  we  consider  that  ADS  supports  dynamic 
simulation  and  joins  and  retires  ,  the  V&V  problem  becomes  impractical  to  define,  let  alone  to 
satisfy.  We  hastened  to^add  that  this  does  not  automatically  condemn  the  concept.  It  simply 
asserts  that  W&A  has  to  be  rethought. 

Although  many  of  the  technical  support  requirements  named  on  the  slide  are 
already  a  part  of  the  DMSO  HLA  build  effort,  we  asserted  them  to  ensure  that  they  are  stated 
matters  of  record,  in  case  resource  or  technical  constraints  demand  reduction  in  HLA  scope. 
Specifically,  we  saw  the  need  for  the  following: 

The  HLA  must  define  specifications  for  the  sharing  of  object  attributes  and 
methods  (attributes  and  behavior)  to  facilitate  consistent  object  behavior  across  heterogeneous 
simulations. 

The  HLA  must  define  a  method  for  the  exchange  of  event  context  information. 
By  this  we  mean  that  triggering  events  must  be  visible  with  each  observable  event.  For 
example,  a  Stinger  launch  event  PDU  should  include  reference  or  mapping  to  the  event  which 
justified  the  launch  event,  like  an  aircraft  detection.  Although  analysts  need  to  define  this 
clearly  for  the  technicians  to  incorporate  in  the  HLA  development,  this  capability  was 
considered  essential  to  meaningful  distributed  analysis. 

The  RTI  of  the  HLA  must  continue  to  include  variable  time  control,  and  must 
develop  switches  which  support  the  pre-runtime  analytic  setup  and  data  capture  filters,  as  well 
as  run-time,  real  time,  data  analysis  and  capture  adjustments. 
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Relevance  of  ADS:  Issues  in  Integrating  the  Acquisition  Process  (Continued) 

•  W&A  policy  and  practices 

•  Simulation  architecture  specifications 

»  Shared  attributes  and  methods 
»  Shared  contextual  information 

•  Common  shared  environment 

-  Across  multiple  physical  domains  and  spectra 

•  HLA's'RTI  specification  must  support 

-  Variable  time  control  (R/T,  FTRT,  start/stop/restart) 

-  Analytic  setup  and  data  capture 

-  Setup  and  run-time  data  analysis  and  capture  tools 

•  Requirements  to  support  people: 

-  Collaborative  tools  (parallel  network:  GroupWare.VTC) 

-  Education/training  of  participants 

•  *  Management  process  change  required;  culmral  acceptance  may  be  a  problem 
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A  fundamental  benefit  of  the  integrated  acquisition  process  shown  on  the 
previous  slide,  is  diat  it  enables  collaboration  among  a  variety  of  communities,  most  of  whom  do 
not  communicate  with  each  other  today.  Technology  to  support  this  human  collaboration  is  just 
as  important  as  technology  to  support  the  object  library  and  simulations.  This  includes  a 
parallel  network  which  would  support  collaborative  analysis  using  modem  collaboration  tools 
like  GroupWare,  and  VTC.  Without  this,  ADS-A  in  support  of  the  acquisition  effort  and 
probably  in  every  other  functional  area  as  well,  would  not  attain  the  level  of  effectiveness 
needed  to  make  it  worth  the  expense  of  attempting  it. 


Finally,  we  dealt  with  what  will  probably  be  the  hardest  issue  of  all:  cultural 
reluctance  and  people  management.  Integrating  the  acquisition  process  will  entail  adopting  an 
integrated  process  management  philosophy.  This  kind  of  dramatic  change  will  not  be  readily 
adopted  by  all  affected  parties.  In  addition,  the  sheer  simulation  complexity  implied  in  ADS 
mandates  a  fresh  look  at  the  process  of  educating  and  monitoring  the  assignments  of  the  military 
analyst  population.  Tlie  prevailing  approach  of  sideline  tours  in  staff  analyst  positions  is 
incompatible  with  the  time  required  to  attain  familiarity  and  comfort  with  ADS  technologies. 
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WG  5:  Requirements  for  Training  Analysis 

•  Relevance  of  ADS 

-  Training  standards  definition  (analysis  provides  feedback  —  ID  deficiencies) 

-  Scenario  generation 

»  Bring  in  additional  collaborators 
»  Pre-exercise  Scenario  validation 
~  Exercise  Conduct 

»  Improved  interaction  with  trainer 

-  Post-exercise  AAR  and  analysis 

»  Supports  improved  event  understanding 

•  Issues 

-  Inadequate  data  collection 

»  Contextual  information  (context  for  exercise  and  cause/effect  trace) 

»  Voice  digitization 
»  Visual/video  capture 

-  Incomplete  definition  of  MOEs/MOPs 

»  Need  to  move  beyond  K/W  scoreboard 

-  Inadequate  data  correlation  tools  for  distributed  environment 
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This  slide  discusses  the  potential  relevance  of  ADSA  to  the  four  phases  of 
training  and  identifies  areas  in  which  improvements  are  needed. 

1.  Definition  of  Training  Standards.  The  Services  define  the  tasks  which  are  to  be  trained  and 
the  standards  to  which  units  (or  individuals)  must  train  each  task.  Analysis  using  ADS  can 
provide  feedback  to  the  training  community  to  help  identify  training  deficiencies.  For  example, 
if  unit  trmning  histories  and  training  outcomes  were  saved  (ignoring  the  anonymity  issue  for  a 
moment),  groups  of  outcomes  associated  with  a  particular  training  event  could  be  analyzed  to 
identify  distributions  that  seem  irregular;  then  the  training  support  packages  associated  with  the 
event  which  might  be  modified. 

2.  Scenario  Generation.  A  distributed  environment  can  enrich  the  development  of  scenarios  for 
training  by  enabling  disparate  communities  to  collaborate.  Currently,  most  training  exercises 
begin  with  scripted  orders  and  force  laydowns  being  given  to  the  units.  In  the  future, 
intelligence  an^ysts,  as  an  example,  could  participate  by  providing  realistic  information  through 
the  distributed  environment  to  be  used  in  the  intelligence  preparation  of  the  battlefield.  Another 
use  for  ADS  in  this  area  is  to  review  and  test  the  scenario  in  a  simulated  environment  prior  to 
using  it  in  a  training  environment. 

3.  Exercise  Conduct.  ADS  supports  an  unprecedented  degree  of  non-invasive  analyst 
interaction  witih  the  trainer  during  the  training  evolution.  In  addition  to  providing  contextual 
insight  to  the  progress  of  the  evolution,  ADS  here  affords  the  analyst  the  chance  to  refine  the 
collection  efforts  in  real  time. 

4.  Post-Exercise  AAR  and  Analysis.  ADS  supports  substantively  improved  situational 
awareness  through  distributed  visualization.  The  capability  of  replaying  selected  pieces  of  an 
evolution  from  varying  points  on  the  battlefield  or  from  varying  perspectives  in  the  chain  of 
command  which  will  provide  insight  and  data  not  available  before. 
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WG  5:  Requirements  for  Training  Analysis  (Continued) 

•  Relevance  of  ADS 

-  Training  standards  definition  (analysis  provides  feedback  —  ID  deficiencies) 

-  Scenario  generation 

»  Bring  in  additional  collaborators 
»  Pre-exercise  Scenario  validation 

-  Exercise  conduct 

»  Improved  interaction  with  trainer 
“  Post-exercise  AAR  and  analysis 

»  Supports  improved  event  understanding 

•  Issues 

-  Inadequate  data  collection 

»  Contextual  information  (context  for  exercise  and  cause/effect  trace) 

»  Voice  digitization 
»  Visual/video  capture 

-  Incomplete  definition  of  MOEs/MOPs 

»  Need  to  move  beyond  K/V  scoreboard 

-  Inadequate  data  correlation  tools  for  distributed  environment 


0$D/PA&B 


Though  there  is  much  fertile  ground  in  analyzing  training  exercises,  either  as 
AARs  for  the  unit  or  for  other  purposes,  our  analytic  capabilities  are  hampered  by  the  following 
issues: 

1.  First,  there  is  not  enough  data  captured  to  support  a  complete  analysis. 

Most  importantly,  contextual  information  is  generally  not  available  and  if  it  is  available,  it  is 
normally  not  in  a  digital  format.  Contextual  information  describes  the  purpose  of  the  exercise 
and  its  meaning;  this  provides  the  context  for  understanding  what  the  exercise  was  trying  to 
achieve.  (This  is  a  more  general  problem  with  ADS  that  is  not  confined  to  training.) 

Analysis  of  C2  decisions  and  their  effects  is  an  area  of  rich  potential  for 
analysts  but  is  relatively  uncharted.  (See  Army  Research  Institute-Ft.  Knox  work  to  assess  the 
value  of  digitized  C2  equipment  (POSNAV,  IVIS)  as  a  significant  exception.)  Until  training 
exercises  incorporate  digital  C2,  much  (and  sometimes  all)  of  the  C2  information  will  be  in 
voice  radio  traffic.  This  information  is  very  seldom  recorded  and  even  when  it  is,  there  do  not 
seem  to  be  any  commercially  available  tools  to  support  word  spotting  or  speech  processing  .  A 
complicating  factor  is  the  generally  poor  quality  of  these  recordings.  Voice  transmissions 
should  be  digitized  and  parsed  for  evolution-significant  events. 

Video  and  visual  information  that  may  be  available  to  the  trainer  is  not 
currently  captured  in  formats  which  facilitate  analysis.  Video  capture  digitization  needs  to  be 
indexed  to  event  time  so  that  the  analyst  can  collaborate  visually  those  events  for  which  other 
recorded  data  is  confusing  or  contradictory. 

2.  Second,  the  kinds  of  MOPs  that  the  analytic  community  currently  relies  on 
are  at  the  killer-victim  scoreboard  level.  These  MOPs  describe  the  facts  of  the  battle,  but  do  not 
shed  light  on  the  decisions  made  and  the  reasons  for  those  decisions.  We  need  to  move  to  more 
abstract  MOPs,  and  to  MOPs  that  describe  cause  and  effect  relationships. 

3.  Third,  given  a  variety  of  MOPs  and  the  complexity  of  cross-simulation  data 
correlation,  the  analyst  would  like  to  know  which  factors  are  conelated  with  each  other.  We 
need  better  tools  for  this. 
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WG  5  Summary:  BROAD  THEMES 

•  Resolution: 

-  More  is  not  better  for  all  uses 

-  Needs  to  be  appropriate  for  application 

•  Major,  largely  unrecognized  benefit  to  ADS,  is  opportunity  for  collaborative  analysis 

-  Does  this  need  specific  kinds  of  tools  to  support  it? 

•  Technical  Issues  in  Support  of  ADSA 

-  Improved  data  availability  and  collection 

-  Ability  to  capture  contextual  information,  better  (higher-level)  MOPs,  better  analysis  tools 

-  Tools  for  Distributed  Exercise  Management  (DEM)  and  setup(AMG?) 

-  Emerging  simulation  standards  need  to  accommodate  ADSA  requirements 

•  Sidebar:  Analyst  training  and  education  issue 

-  College  curriculum 

-  Complexity  of  subject  matter 

-  Educating  Decision  makers 
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We  summarize  important  observations  from  the  preceding  discussions  here: 

1.  Analysis  does  not  always  benefit  from  higher  resolution.  In  fact,  high  resolution 
adds  unwanted  complexity  to  some  applications  —  complexity  that  makes  execution  difficult  and  may 
actually  confound  results.  More  appropriate  is  an  integrated  hierarchical  farmly  of  models  so  that  the 
detail  of  high  resolution  ADS,  especially  including  human  interaction  effects,  may  be  used  to  calibrate 
lower  resolution  models  or  to  help  generate  aggregate  data. 

2.  Analysts  have  always  collaborated  —  over  structure,  data,  models  and  the 
analysis  itself.  Collaboration  has  always  been  difficult  if  the  participants  were  not  co-located,  but  it  is 
now  more  difficult  in  ADS  because  the  overall  process  is  distributed.  That  need  not  remain  the  case  if 
we  design  and  provide  a  parallel  “network”  to  enable  analyst  collaboration.  Regardless  of  whether  the 
models  or  simulations  are  widely  distributed,  the  process  of  analysis  can  be  widely  distributed, 
enabling  collaboration  and  providing  important  leverage  to  Defense  analysis, 

3.  To  enhance  or  enable  the  use  of  ADS  for  analysis  requires  investment  in 
appropriate  tools  and  capabilities: 

•  First  and  foremost,  valid  data  must  be  made  widely  available  at  all  levels. 

At  the  aggregate  levels,  much  work  remains  to  be  done  to  support  future 

models  such  as  JWARS.  The  problems  with  providing  higher  resolution 

data  are  well  known. 

•  The  nature  of  the  data  collected  must  also  be  enhanced  to  allow  addressing 
details  inherent  in  some  issues.  For  example,  data  which  captures  the  context  of 
simulation  events  is  necessary  to  allow  causative  analysis.  This  data  is  now 
gathered  apart  from  the  automated  data  and  is  hard  to  store  and  use.  As 
important,  are  the  tools  necessary  to  enable  data  analysis.  This  includes  the 
ability  to  handle  large  amounts  of  data  distributed  at  several  locations  and  the 
ability  to  do  sophisticated  data  analysis  and  correlation. 
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WG  5  Summary:  BROAD  THEMES  (Continued) 


•  Resolution: 

-  More  is  not  better  for  all  uses 

-  Needs  to  be  appropriate  for  application 

•  Major,  largely  unrecognized  benefit  to  ADS,  is  opportunity  for  collaborative  analysis 

-  Does  this  need  specific  kinds  of  tools  to  support  it? 

•  Technical  Issues  in  Support  of  ADS-A 

-  Improved  data  availability  and  collection 

-  Ability  to  capture  contextual  information,  better  (higher-level)  MOPs,  better  analysis  tools 

-  Tools  for  Distributed  Exercise  Management  (DEM)  and  setup  (AMG  issue?) 

-  Emerging  simulation  standards  need  to  accommodate  ADS-A  requirements 

•  Sidebar:  analyst  training  and  education  issue 

-  ..College  curriculum 

-  Complexity  of  subject  matter 

-  Educating  decision  makers 


OSD/PA&E  OBSSSB&SI 


•  Finally,  the  overall  process  of  exercise  execution  must  be  made  simpler. 
The  time  necessary  to  assemble  an  analysis  should  be  substantially  reduced 
by  providing  tools  to  do  the  following: 

-  Assist  with  scheduling  and  coordinating  resources 

-  Developing  and  documenting  scenarios 

-  Pre-exercise  evaluation  relating  issues  to  experimental 
design,  MOEs  and  data  adequacy 


4.  The  final  observation  is  that  providing  these  technical  capabilities  will  not 
fully  answer  the  issues  of  ADS-based  analysis.  We  believe  that  the  nature  of  analysis  will  be 
fundamentally  more  demanding.  Technology  can  make  using  the  environment  feasible,  but  the 
complexity  of  the  environment  will  demand  better  preparation  of  the  analyst  and  the  decision 
maker.  The  analyst  will  require  better  understanding  of  the  distributed  environment  itself, 
implying  better  education  in  computer  science.  And  analytic  complexity  issues  imply  that  the 
analyst  must  be  able  to  analyze  the  analysis  beforehand  for  complexity,  resource  requirements 
and  appropriate  analytic  approach.  Likewise,  decision  makers  must  understand  the  issues 
required  by  ADS  analysis,  both  in  terms  of  cost  to  execute  and  the  quality  and  substance  of  the 
analytic  results. 
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MORS  ADSA  Workshop 

WG  Scope:  “Address  Broad  Analytic  Questions  within  [your]  Domain” 

ref:  Terms  of  RcfereiKc 


•  When  should  ADS-based  capabilities  be  used? 

-  When  analytic  complexity  from  added  detail  is  necessary,  affordable  and  appropriate. 

•  Are  ADS-based  capabilities  better  or  worse  for  specific  types  of  analysis? 

-  This  is  specific  to  the  application,  but  it  is  also  very  useful  for  distributed  collaborative  analysis. 

•  Does  ADS  allow  us  to  address  any  new,  necessary  analytic  questions? 

-  For  operations,  OOTW  and  C4ISR:  For  acquisition,  integrated  system  mgmt/evaluation. 

•  How  do  ADS-based  capabilities  fit  with  other  types  of  analytic  capabilities  and  approaches? 

-  By  applying  direct  application  to  acquisition  and  indirect  for  force  structure  analysis.  Provides  a  medium 
for  peer  review  throu^  distributed  collaborative  analysis. 

•  What  are  limitations  of  current  and  future  ADS-based  capabilities? 

-  Current  data  content,  storage  and  analysis  must  be  overcome  in  general 

-  Current  and  future,  analysis  is  more  complex,  time  consuming,  needs  more  preparation. 

•  Tools  and  education  lighten  impact,  but  increased  complexity  remains  a  limitation. 


These  final  two  slides  provide  short  answer  replies  to  the  nine  questions  originally  provided  to 
the  working  groups  in  the  TOR.  They  are  provided  here  as  a  final  summary  of  the  WG  five 
report. 


MORS  ADSA  Workshop 

WG  Scope:  “Address  Broad  Analytic  Questions  within  [your]  Domain” 

ref:  Terms  of  Reference  (Continued) 


•  Are  there  pitfalls  in  analysis  using  ADS-based  capabilities? 

-  Demand  on  resources,  more  time,  need  for  tools,  complexity  and  a  need  for  collaboration. 

•  Does  the  use  of  ADS>based  capabilities  affect  the  utility  of  analytic  results? 

-  Complexity  may  cause  “anecdotal”  versus  “analytic.”  But,  “immerse”  decision  maker  in  process  and/or 
results. 

•  Are  there  useful  ways  to  utilize  the  results  from  ADS-based  analysis,  exercise  and  training  that  differ  from 
conventional  methods? 

-  Concurrent  or  spiral  development  for  acquisition.  Calibration/validation  for  force  effectiveness. 

•  Do  ADS-based  analysis  capabilities  make  W&  A  processes  any  harder,  easier  or  different'!  But  more 
important! 


OSD/PA&E 


MORS  ADSA  Workshop 

WG  5:  Postscripts  and  Minority  Positions 


•  Definition  of  ADS  (Dr.  Paul  Davis) 

-  For  the  analytic  conununity  the  definition  of  ADS  should  tend  to  be  inclusive 

-  Protocols  should  not  be  required.  A  communication  scheme  linking  models  without  protocols  would  be 
advanced. 

•  Using  ADS  for  Analysis  (Dr.  Paul  Davis) 

-  ADS  “should  be  thought  of  a  mode  of  operations  for  accomplishing  EXPERIMENTS. 

-  BUT  not  as  a  good  device  for  analysis  per  se  " 

-  However,  experiments  can  be  powerjul 

•  Added  ADS  Requirements  for  force-level  effectiveness  (John  Yanaros) 

-  Ability  to  share  data  and  results  in  distributed  fashion 

-  Master  DoD/JS  analytic  exercise  calendar  for  distributed  analysis  (like  training  exercise  calendar) 


OSD/PA&E 
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During  post- Workshop  reviews,  comments  were  returned  which  expanded  upon  what  was  said 
during  the  Workshop  or  represented  a  valuable  minority  position  on  an  issue.  Hence  shorter 
comments  are  annotated  directly  in  the  appropriate  text.  These  comments  are  more  substantial, 
and  therefore  are  noted  separately. 

Dr.  Paul  Davis  stood  out  from  the  crowd  somewhat  by  asserting  that  ADS 
would  never  be  suited  to  analysis  as  we  think  of  it  using  closed,  constructive  models.  Rather,  he 
more  distinctly  suggests  ADS  as  suited  to  experiments,  from  which  insights  are  gained  or  data 
developed  to  help  calibrate  the  constructive  analytic  models.  This  is  rather  a  stronger  opinion 
than  the  Group  discussion,  but  does  follow  the  complexity  and  aggregation  issues  discussed 
previously. 


John  Yanaros  added  a  few  more  requirements  for  ADS-based  analysis.  The 
paradigm  of  collaborative  and  distributed  analysis  will  pose  problems  in  developing  data  and 
distributing  results  to  the  distributed  analysts,  and  the  limitations  on  the  shared  distributed  assets 
will  result  in  a  need  to  manage  the  distributed  analysis  calendar  for  major  events.  Both  of  these 
things  can  be  addressed  during  the  development  of  a  collaborative  analysis  environment  if  we 
pay  attention  up  front. 
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MORS  ADS  Workshop 
Working  Group  6:  Understanding 
Behavior/Performance 


Working  Group  six 


One  of  the  more  important  contributions  of  ADS  is  the  ability  to  use  either 
virtual  or  live  simulations  to  understand  and  capture  human  behavior. 

This  is  an  aspect  of  constructive  simulations  that  has  not  been  well 
represented  in  the  past.  Therefore,  WG  six  is  addressing  an  important 
issue  for  the  analysis  process. 
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The  Mission  of  Working  Group  6 
with  Amended  Objective 


Address  the  nine  questions  from  the  ADS  Workshop  TOR 
list  in  the  context  of  the  WG  six  objective  as  amended. 


Analysis  of  DIS  simulations  with  the  objective  of 
deepening  understanding  of  warrior  behavior/ 
performance  and  then  incorporating  that  understanding 
in  improved  analysis,  constructive  models  and  data. 


Working  Group  six 


We  were  asked  to  address  the  nine  questions  from  the  TOR  list  in  the 
context  of  the  WG  six  objective.  We  felt  the  need  to  amend  the  objective 
as  shown  in  italics.  In  addition  to  incorporating  an  improved 
understanding  of  warrior  behavior  in  constructive  models  and  data,  we 
wished  to  emphasize  that  that  Improved  understanding  should  be 
incorporated  at  the  top  level  of  the  intellectual  process,  the  analysis  itself, 
not  just  the  tools  that  help  execute  the  analysis.  Next,  we  defined  ADS. 
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Definition:  Advanced  Distributed  Simulation 


•  Advanced:  Applies  to  all  training,  readiness  and  the 
requirements  prototyping  process  with  the  same  core 
technology  base 

•  Distributed:  Shared  battlefield  data  are  entered  from 
geographically  separated  sites  via  communications 
networks 

•  Simulation:  Allows  mix  and  match  of  live,  constructive 
and  virtual  simulation  methods 


Working  Group  six 


These  definitions  were  developed  to  provide  us  with  a  framework  for 
further  study  of  the  question. 


Approach:  We  Split  Into  Two  Subgroups 


•  One  subgroup  used  “Seven  Steps  of  Design  for 
Success,”  a  systems  design  approach  to  the  problem 

•  The  second  subgroup  used  a  matrix  approach  to 
compare  steps  in  the  analysis  process  with  questions 
from  ADS  Workshop  TOR 

•  Remarkable  convergence  of  results  from  two  different 
approaches 


Working  Group  six 


We  split  into  two  subgroups.  The  first  used  a  system  called  “Seven 
Steps  of  Design  for  Success,”  a  commercially  supported  systems  design 
approach.  This  could  be  characterized  as  a  synthesis  process. 

The  second  subgroup  compared  the  steps  in  the  analysis  process  with 
the  questions  from  the  workshop  TOR.  This  is  fundamentally  an  analysis 
process. 

As  we  will  see  later  in  the  report,  the  synthesis  process  and  the  analysis 
process  converged  rather  well. 
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General  Chair’s  Guidance  to  ADSA  WG  Chairs: 
Consider  the  Following  Questions 


1.  What  are  the  most  central  requirements  of  needed 
analysis?  (current  and  future) 

2.  Which  models  and  simulations  are  most  frequently 
used  now? 

3.  What  are  key  limitations  of  these  models  and 
simulations  in  meeting  analytical  needs?  (current 
and  future) 

4.  Are  there  planned  developments  to  resolve  these 
limitations? 

5.  Do  ADS-based  tools  offer  the  potential  to  meet 
central  requirements  of  needed  analysis? 

6.  Are  they  different  from  or  redundant  with  other 
approaches  existing  or  planned?  (How?) 


Working  Group  six 


The  general  chair  of  the  Workshop  asked  the  working  groups  to  consider 
these  questions.  We  felt  the  need  to  organize  the  questions  as  we  will 
show  on  subsequent  slides. 


General  Chair’s  Guidance  to  ADSA  WG  Chairs: 
Consider  the  Following  Questions  (Continued) 


7.  Do  the  desired  ADS-based  capabilities  exist?  Have 
they  been  “Verified  and  Validated”?  Is  there  a 
funded  program  in  place,  or  is  there  one  that  could 
be  modified? 

8.  Are  there  challenges  to  conducting  credible  analysis 
using  ADS?  (Are  they  unique  or  harder  than  with 
other  methods?) 

9.  What  are  the  impediments  to  using  ADS-based 
analytical  tools,  and  how  can  they  be  resolved? 
(What  have  demonstration  programs  shown  us  we 
need  more  —  for  what?  Does  the  current  and 
planned  set  of  DIS  PDU’s  address  relevant  areas  of 
analytic  interest  with  sufficient  fidelity?  Is  there 
sufficient  linkage  between  instrumentation  and 
simulations,  etc.?) 

Working  Group  six 


These  questions  are  quite  involved  and  are  in  fact  the  subject  of 
numerous  published  papers.  For  the  convenience  of  the  reader,  we  cite 
the  following:  MR-744-AF,  Understanding  the  Air  Force’s  Capability  to 
Effectively  Appiy  Advanced  Distributed  Simulation  for  Analysis:  An 
Interim  Report,  1996,  T.  Lucas,  B.  Kerchner  and  John  A.  Friel.  This 
report  is  cited  not  because  it  is  particularly  profound,  but  because  it  is 
easily  available  from  RAND  and  includes  an  extensive  bibliography.  This 
would  provide  a  convenient  starting  point  for  someone  interested  in 
becoming  familiar  with  the  subject. 
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Correlation  of  TOR  Questions  with 
Steps  in  Analysis  Process 


Strongly  Correlated  l  Conelated  I--:'  I  Weakly  Correlated 


Working  Group  six 


In  an  attempt  to  provide  a  framework  for  our  discussion,  we  created  this 
matrix  with  the  rows  defined  by  a  representation  of  the  steps  in  the 
analysis  process  and  the  columns  defined  by  the  question  numbers  from 
the  previous  slides.  The  dark  cells  indicate  that  we  believe  that  the 
question  was  strongly  correlated  with  the  given  step  in  the  analysis 
process.  For  example,  we  believe  that  Question  1 ,  “What  are  the  most 
central  requirements  of  needed  analysis?  (current  and  future)”  is  strongly 
correlated  with  the  step  in  the  analysis  process, 
“Select/develop/assemble  tools  and  obtain  data.” 
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TOR  Questions  Focused  on  Particular 
Steps  in  Analysis  Process 


•  Analysis  approach  development,  tool  selection  and 
communication  of  results  most  affected  by  TOR  issues 

•  Issues  raised  by  questions  three,  four,  five  and  nine 
most  important  to  analysis  process 

•  Introduction  of  ADS  does  not  significantly  impact 
problem  definition 

-  Problems  are  not  significantly  different  or  new 

-  ADS  affects  HOW  questions  are  approached 


Working  Group  six 


This  slide  captures  in  words  the  information  contained  in  the  matrix.  The 
first  bullet  refers  to  the  fact  the  second,  third  and  sixth  rows  have  the 
most  black  cells. 

The  second  bullet  refers  to  the  fact  that  columns  three,  four,  five  and  nine 
have  the  most  black  cells. 

The  third  bullet  reflects  that  fact  that  ADS  doesn’t  interact  very  strongly 
with  the  problem  definition  step  of  the  analysis  process.  This  first  row  is 
the  one  with  the  least  overall  correlation. 
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Two  Subgroups  Mapped  Nine  TOR  Questions  into 
Four  Aggregate  Questions 


Question  A: 
Question  B: 
Question  C: 
Question  D: 


Aggregate  of  Questions  1-4 
Aggregate  of  Questions  5-6 
Aggregate  of  Questions  7-8 
Question  9 


Working  Group  six 


To  organize  the  results  of  our  discussions,  we  aggregated  the  nine 
questions  from  the  TOR  into  four  aggregated  questions,  as  shown. 
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Question  A:  ADS-Based  Capabilities 


When  should  ADS-based  capabilities  be  used;  are  they  better 
or  worse  for  specific  types  of  current  analysis;  do  they  fit  with 
other  analytic  capabilities;  and  do  they  allow  the  analyst  to 
address  any  new,  necessary  analytic  questions? 

WHEN 

•  Less  cost,  risk  and  environmental  impact  than  live 
exercises  and  tests 

•  New  CONOPs,  technologies,  systems  and 
organizational  structures  are  being  evaluated 

Working  Group  six 


Question  A,  aggregated  from  questions  1  through  4,  is  shown  at  the  top 
of  the  slide.  The  answers  are  shown  under  the  general  heading  “When. 

The  first  bullet  is  an  obvious  cost-effectiveness  consideration.  The 
second  bullet  involves  developing  new  ways  for  HITL  to  interact  with 
systems  and  with  each  other. 


Question  A:  ADS-Based  Capabilities  (Continued) 


WHEN  (Continued) 

•  HIL  more  viable,  acceptable  and  valid 

•  Lack  of  constructive  alternatives 

•  Assess  impact  of  information  on  force  effectiveness 

•  Involvement  of  decision  maker  desired 

•  Demonstration  of  results  desired 

•  Assembly  of  simulation  components  at  one  location 
impractical 


Working  Group  six 


The  first  four  bullets  under  the  heading  “When”  involve  the  importance  of 
the  role  of  HITL  in  the  analysis.  All  of  these  considerations  suggest  a 
need  to  include  virtual  or  live  simulations  in  the  ADS  approach  to  an 
analysis. 

Demonstration  of  results  is  currently  one  of  the  strong  points  of  ADS.  It 
is  possible  to  do  very  illuminating  demonstrations  of  system  behavior  and 
results  in  general.  Some  traditional  analysts  argue  that  the  compelling 
nature  of  demonstrations  available  through  ADS  often  masks 
fundamentally  weak  analysis. 

Obviously  ADS,  with  emphasis  on  the  Distributed,  would  be  attractive 
when  all  the  simulation  components  are  not  available  at  one  location. 


109 


Question  A:  ADS-Based  Capabilities  (Continued) 


SOLUTION  SPACE 

ADS  is  not  always  the  appropriate  tool  to  be  used  for 
analysis.  DMSO  and  the  appropriate  Service  agencies 
need  to  provide  guidelines  for  when  and  how  to  use  ADS 


Working  Group  six 


ADS  is  not  always  the  answer.  DMSO  and  the  appropriate  Service 
agencies  should  provide  guidance  on  this  subject.  We  refer  back  to  the 
two  previously  cited  documents  as  a  source  of  the  views  that  exist  in  the 
analysis  community  on  the  subject  of  ADS. 
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Question  B:  Current  Limitations 
and  Future  Pitfalls 


What  are  the  limitations  of  current  and  future  ADS-based 
capabilities,  and  what  are  the  pitfalls  in  analysis  using 
these  capabilities? 

LIMITATIONS 

•  Reliability  of  federated  simulations 

•  Lack  of  reproducibOity 

•  Limited  number  of  replications 


Working  Group  six 


Question  B  is  an  aggregation  of  questions  five  and  six  from  the  TOR. 
The  “limitations”  are  rather  obvious  implications  of  using  ADS. 


Question  B:  Current  Limitations 
and  Future  Pitfalls  (Continued) 


LIMITATIONS  (Continued) 

•  Cost,  coordination  and  competing  resources  required 

•  Data  security  (multiple  classification  sources,  multiple 
levels  and  proprietary) 

PITFALLS 

•  ‘‘Gaming  the  game!’’ 

•  ADS  as  “only  approach  to  analysts”  syndrome 

•  Failure  to  control  human  factors 

•  High  visibility  —  institutional  bias  for  “RIGHT 
answer” 

Working  Group  six 


The  two  “Limitations”  cited  are  not  uniquely  related  to  ADS,  but  rather  are 
problems  that  exist  whenever  large  scale  analysis  is  done. 

The  first  and  third  “Pitfalls”  are  a  consequence  of  any  experimentation 
that  involves  humans.  The  second  and  fourth  “Pitfalls”  are  a 
consequence  of  the  current  popularity  of  the  ADS  technique. 
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Question  B:  Current  Limitations 
and  Future  Pitfalls  (Continued) 


SOLUTION  SPACE 

•  Central  repository  needs  to  be  developed  for  analysts 
for  use  to  coordinate  schedules  of  ADS  facilities  and  to 
provide  information  about  resources  available  at  those 
facilities,  including  simulators,  software,  expertise,  data 
bases,  etc. 

•  Investment  in  ADS  needs  to  be  rationalized  and 
coordinated 


Working  Group  six 


Certainly,  an  easily  accessible  database  containing  information  on  ADS 
facilities,  schedules  and  the  like  would  be  a  very  valuable  asset  to  the 
community. 

The  additional  costs  associated  with  using  ADS  have  to  be  understood 
up  front,  and  the  investments  in  facilities  and  software  need  to  be 
coordinated. 
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Question  C:  Utility  of  Analytic  Results 
and  New  Ways  to  Utilize  These 


Do  ADS-based  capabilities  affect  the  utility  of  analytic 
results,  and  are  there  useful  ways  to  utilize  these  results 
that  differ  from  conventional  methods? 

•  Use  ADS-based  results  to  calibrate  and  validate 
conventional  methods 

•  Ability  to  develop  HITL  data  repository 

•  Operators  in  the  loop 

•  Decision  makers  in  the  loop 


Working  Group  six 


ADS  can  provide  valuable  insight  into  how  HITL  should  be  represented  in 
constructive  models.  An  HITL  data  repository  could  be  developed  to 
calibrate  the  kinds  of  tasks  that  are  representative  of  military  applications 
and  the  time  it  takes  to  execute  those  tasks.  ADS-based  capabilities 
would  dramatically  improve  our  ability  to  represent  operators  and 
decision  makers  in  the  loop. 
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Question  C:  Utility  of  Analytic  Results 
and  New  Ways  to  Utilize  These  (Continued) 


•  Use  stakeholders  in  the  loop  to  enhance  utility  of  results 

•  Easier  to  communicate  results  to  non-participants 

SOLUTION  SPACE 

•  Resources  should  be  provided  to  assist  with  training 
analysts  in  the  use  of  ADS  —  for  example,  courses  and 
training  materials 

•  Promote  the  collection  and  collation  of  the  human 
behavior/performance  data  from  planned  ADS  exercises 
for  use  in  the  calibration  of  constructive  models 

Working  Group  six 


Stakeholders  in  the  loop  of  an  ADS  exercise  could  enhance  the 
acceptability  of  the  analytic  results  in  the  operational  community.  But 
ADS  also  has  an  inherent  capability  to  communicate  the  richness  of  the 
results  to  non-participants. 

Serious  attempts  must  be  made  to  train  analysts  in  the  use  of  ADS  in 
academic  institutions,  such  as  the  Air  Force  Institute  of  Technology 
(AFIT)and  the  Naval  Post  Graduate  School  (NPS).  In  addition,  senior 
service  schools  should  attempt  to  make  decision  makers  aware  of  when 
ADS  is  an  appropriate  technique  to  employ  and  how  to  interpret  the 
results  of  ADS  exercises. 

ADS  exercises  should  be  planned  so  as  to  provide  inputs  to  a  database 
of  human  behavior  and  performance  for  use  in  calibrating  constructive 
models. 


Question  D:  Verification,  Validation, 
and  Accreditation  (VV&A) 

Do  ADS-based  analysis  capabilities  make  the  W&A  process 
any  harder,  easier  or  different? 

•  Face  validity  is  easier 

•  Other  forms  of  validity  are  harder 

•  Configuration  control  is  harder 

•  Operator  or  decision  maker  in  the  loop  makes 
accreditation  easier 

SOLUTION  SPACE 

•  Continue  emphasis  on  V&V  methodology  in  each  Service 


Working  Group  six 


W&A  are  difficult  enough  when  only  one  constructive  model  is  present. 
Because  of  the  enhanced  graphics  normally  associated  with  ADS,  face 
validity  is  easier.  But  other  forms  of  validity  are  harder  because  of  the 
multiplicity  of  simulations  and  locations.  Similarly,  configuration  control  is 
harder.  However,  operators  and  decision  makers  in  the  loop  may  not 
help  with  V&V,  but  accreditation  may  be  an  easier  status  to  achieve. 

A  solution  to  this  question  will  only  come  through  continued  hard  work  on 
the  part  of  the  analytic  communities  of  the  Services. 
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This  flow  chart  provides  a  logic  flow  for  deciding  upon  the  “mix”  of 
simulations  to  address  an  issue.  Note  that  the  process  selects  a 
constructive  simulation  as  a  first  choice.  If  one  is  adequate,  virtual  and 
live  simulations  are  not  considered.  Similarly,  if  an  adequate  combination 
of  constructive  and  virtual  simulations  can  be  found,  live  simulations  are 
not  considered.  This  probably  represents  the  current  attitude  of  the  DoD 
analytic  community. 
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Summary  of  Recommendations 


•  ADS  is  not  always  the  appropriate  tool  to  be  used 
for  analysis.  DMSO  and  the  appropriate  Service 
agencies  need  to  provide  guidelines  for  when  and 
how  to  use  ADS.  (Question  A) 

•  A  central  repository  needs  to  be  developed  for 
analysts  for  use  to  coordinate  schedules  of  ADS 
facilities  and  to  provide  information  about 
resources  available  at  those  facilities,  including 
simulators,  software,  expertise,  data  bases,  etc. 
(Question  B) 

Working  Group  six 


ADS  is  not  always  the  preferred  approach  to  an  analysis.  Better  OSD 
and  service  guidelines  should  be  published  on  this  issue 

A  central,  easily  accessed  database  describing  ADS  facilities,  simulators, 
software,  databases  and  experts  should  be  developed. 
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Summary  of  Recommendations  (Continued) 


•  Investment  in  ADS  needs  to  be  rationalized  and 
coordinated  within  DoD.  (Question  B) 

•  Resources  should  be  provided  to  assist  with  training 
analysts  and  decision  makers  in  the  use  of  ADS.  For 
example,  courses  and  training  materials  are  needed. 
(Question  C) 

•  Promote  the  collection  and  collation  of  the  human 
behavior  and  performance  data  from  planned  ADS 
exercises  for  use  in  calibration  of  constructive  models. 
(Question  C) 

Working  Group  six 


These  summary  recommendations  come  from  the  deliberations  of  each 
of  the  two  subgroups.  First,  the  investment  in  ADS  should  be 
rationalized  and  coordinated  throughout  DoD.  Resources  should  be 
provided  to  train  analysts  to  perform  ADS  and  decision  makers  to 
understand  when  ADS  is  an  appropriate  approach  to  a  problem. 

The  human  behavior  and  performance  data  that  are  collected  in  ADS 
experiments  must  be  collated  in  an  easily  accessible  database  for  use  In 
calibration  of  constructive  models. 
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Finally,  VV&A  of  ADS  must  receive  renewed  emphasis  within  DoD  and 
the  services. 
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Advanced  Distributed  Simulations  for 
Analysis 
(ADSA  ‘96) 


Synthesis  Panel  Report 

21  February  1996 


Agenda:  Synthesis  Panel 
1  0  Introduction 

0  Nature  erf  the  Pr<rf)lem 
0  Firiding^ecbinniendations 
Q  /l^sessiinent  of  Workshop 


Agenda 


The  Synthesis  Panel  report  is  divided  into  four  sections.  The  first  section, 
Introduction,  describes  the  composition  of  the  Synthesis  Panel  and  its 
concept  of  operations.  The  next  section  characterizes  the  nature  of  the 
problem  as  derived  from  the  deliberations  of  the  Workshop  panels.  It 
discusses  the  definition  of  ADS,  what  is  new  (from  an  analyst's  perspective), 
why  analysts  have  not  embraced  ADS  and  why  analysts  should  bother  with 
ADS.  The  third  section  identifies  and  discusses  the  findings  and 
recommendations  in  four  overarching  areas  that  the  Synthesis  Panel  felt 
were  of  particular  significance.  These  areas  include  the  definition  of  ADS^ 
community  leadership,  an  ADS  plan  for  the  analysis  community  and  ADS 
education,  training  and  experience.  This  report  concludes  with  a  qualitative 
assessment  of  the  Workshop.  It  heuristically  evaluates  the  degree  to  which 
the  participants  were  able  to  satisfy  the  four  objectives  set  forth  in  the 
Workshop  TOR. 


Synthesis  Panel  Members 

0  chair:  Stuart  Starr,  MITRE 

0  Cb"Chatr:  Dr.:Dale  Pace,  JHU/APL(Analyisis  arid 
Requirements) 

0  F^nel  Members: 

-  Robert  Worley;  IDA  (Battlefield  Effectiveness) ; 

-  Dr.  Jerry  Kotchka,  McDonriell  Douglas  (Materiel,  Systems 
and  Acquisition} 

-  Prisciila  Glasow,  SAIC  (T&E) 

-  Howard  Carpenter,  MITRE  (Training,  Mission  Rehearsai 
and  Alternative  courses  of  Action) 

-  Gene  Visco,  FS,  ODUSA(OR)  (Understanding 
Behavior/Performance) 

-  Dick  Hayes,  EBR  (Understanding  Behavior/Performance) 

-  Jack  Thorpe,  SAIC  (floater) 

-  Veni  Bettencourt,  FS,  ODUSA(OR)  (fioater) 

Synthesis  Panel  Members 

This  slide  identifies  the  individuals  that  participated  on  the  Synthesis  Panel. 
In  most  cases  the  individuals  had  dual  assignments.  They  participated  as 
active  members  of  one  of  the  other  six  panels  of  the  Workshop  as  well  as 
the  Synthesis  Panel.  In  selected  cases,  Synthesis  Panelists  “floated"  (i.e., 
they  sampled  the  deliberations  of  several  other  panels  during  the 
Workshop). 

To  focus  the  data  collection  efforts  of  the  Synthesis  Panel,  a  "war  room"  was 
created.  Butcher  block  paper  was  mounted  on  the  walls  of  the  room  and  the 
panelists  posted  summaries  characterizing  the  deliberations  of  the 
Workshop  attendees  as  they  addressed  the  issues  identified  in  the  TOR.  In 
addition,  butcher  block  paper  was  used  to  capture  comments  on  definitions, 
observations  by  plenary  speakers,  insightful  observations,  findings  and 
recommendations  (e.g.,  "blinding  flashes  of  the  obvious"). 

In  addition  to  these  data  collection  activities,  the  Synthesis  Panel  convened 
periodically  after  other  panels  had  adjourned  to  identify  and  discuss  the 
status  of  the  Workshop  and  to  identify  and  discuss  key,  cross-cutting  issues. 
This  report  represents  a  synthesis  of  all  of  the  above  activities. 


What  in  the  World  is  ADS? 


“Gome,  let  us  go  down  and  there  confuse  their  language 
that  they  ntay  not  understand  one  another’s  speech” 

The  Tower  of  Babel 
/  Genesis  11 :7 
;::  (cited  :by  :Sil«>ra  &  Goosey 
PHALANX,  June  1995) 
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What  in  the  World  is  ADS? 

One  of  the  first  questions  that  emerged  during  the  deliberations  was  the 
issue  of  the  definition  of  the  term  ADS.  The  importance  of  developing  a 
definition  that  was  widely  accepted  by  the  community  was  suggested  by  one 
of  the  papers  in  the  read-ahead  package,  “What  in  the  World  is  ADS?"  by 
Sikora  and  Goose.  By  invoking  the  image  of  the  Tower  of  Babel  in  their 
paper,  the  authors  suggested  the  confusion  that  can  ensue  if  the  participants 
in  the  dialogue  lack  a  common  frame  of  reference  and  vocabulary. 

In  recognition  of  this  issue,  the  Synthesis  Panel  spent  a  moderate  amount  of 
time  exploring  the  question,  as  did  many  of  the  other  panels.  It  concluded 
that  the  strawman  definition  in  DoD  5000.59-M  is  inadequate  and  noted  that 
a  taxonomic  approach  to  the  question  shows  some  promise.  That  approach 
is  discussed  at  greater  length  in  the  next  section  of  this  report. 


What’s  New  (from  the  Analyst’s  Perspective)? _ 

0  Analysts  are  being  asked  to 

-  Address  questionsXsome  of  which  are  new)  in  a  more 
sophisticated  way;  e.g., 

>►  E>eal  witha  much  more  complex  world  ambiguous  threats,  ‘^multiple 

»  Address  Issues  from  a  jbint/coalitibn  perspective 
>>4reatC2  issues  morecredibly^  ^^^^^^^^^^^  ^^^  ^  ^ 

5>  C6nslderenvlronmental  effects  more  specifically  (e.g:,  terrain,  obscurants  and 
chemical/biolo^calagents) 

-  Communicate  information  more  effectively  to  the  decision 
:  ^maker;  e.g., ' . 

»  Facintate ‘What  tf ...”  d^^^ 

0  Other  conriniunities  (e;g.,  operations)  are  acquiring 

new  tools  (featuring  Hny,  data  that  may  prove  to^te 
increasing  value  to  analysts 
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What's  New  (from  the  Analyst's  Perspective)? 

Several  of  the  speakers  in  the  plenary  session  (most  notably  the  keynote 
speaker.  Dr.  Anita  Jones,  DDR&E)  observed  that  it  was  propitious  to  conduct 
an  ADSA  workshop  since  there  are  several  things  that  are  new  in  the  world  of 
the  analyst. 

First,  the  analyst  is  being  asked  to  address  questions,  some  of  which  are  new, 
in  a  more  sophisticated  way.  For  example,  with  the  dissolution  of  the  Warsaw 
Pact  and  the  Soviet  Union,  the  world  is  proving  to  b&much  more  complex 
from  a  national  security  perspective  (e.g.,  the  U.S.  faces  a  much  more 
ambiguous  threat;  defense  planning  revolves  around  "multiple  scenario" 
problems;  the  US  military  and  its  allies  are  being  asked  to  undertake  new 
types  of  military  operations,  such  as  complex  humanitarian  operations  in 
Africa  and  peacekeeping  in  the  Former  Yugoslavia).  Since  most  of  these 
issues  involve  complex  joint/coalition  activities,  the  analyst  must  consider  the 
effect  of  joint/coalition  involvement  at  the  very  outset  of  the  analysis.  In 
addition,  it  was  not  unusual,  several  years  ago,  for  analysts  to  assume 
"perfect"  C2  and  to  perform  assessments  within  that  context.  In  view  of  the 
importance  of  C2  in  the  Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS)  Vision 
2010,  such  an  assumption  is  clearly  no  longer  acceptable. 


What’s  New  (from  the  Analyst’s  Perspective?)  (Continued) 

Finally,  there  is  increasing  recognition  of  the  importance  that  a  broad  set  of 
environmental  effects  can  have  on  analysis  (e.g.,  more  granular  terrain  to 
capture  the  effects  of  terrain  masking;  the  properties  of  obscurants  and  their 
effects  on  the  performance  of  Intelligence,  Surveillance  and  Reconnaissance 
(ISR)  systems;  the  challenges  posed  by  detecting,  characterizing  and 
reacting  to  chemical  and  biological  agents). 

Second,  the  analyst  is  being  asked  to  communicate  information  more 
effectively  to  decision  makers.  Most  decision  makers  are  no  longer  satisfied 
with  columns  of  numbers.  They  want  visualization/animation  tools  to 
capture  the  salient  aspects  of  an  analysis  and  the  opportunity  to  participate 
in  a  dynamic,  "what  if..."  dialogue  with  the  analyst. 

Finally,  it  was  observed  that  several  communities  (e.g.,  trainers,  acquirers  of 
systems)  are  acquiring  new  tools  and  data,  both  of  which  have  the  potential 
to  support  analysts  substantively.  This  suggests  the  desirability  of  working 
more  closely  with  these  communities,  both  to  leverage  their  investments  and 
to  share  the  tools  and  data  developed  by  the  analytical  community. 
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Why  Have  Analysts  Not  Embraced  ADS? 

0  Quality  Issues 

-  Ambiguity  about  what  ADS  Is  (and  is  hot) 

-  Replicability  of  results  (e  g.,  dealing  with  human  subjects) 

-  Credibility  Of  results  (e.g.,  W&A  issues) 

-  Suitability  for  issues  confronting  analysts 
0  Use  Issues 

-  Cost/resource  implications 

»  Expensive  to  acquire,  upcfate  and  employ 
V  Difficult  to  get  trained  subjects 

-  Timi^schedule  implications;  time  required  to: 

Create  ADS  capability 
»  Set-up  conditions  for  a  specific  analysis 
»  Perform  test  runs 
>  Reduce  data 


Why  Have  Analysts  Not  Embraced  ADS? 

Several  of  the  panels  spent  considerable  time  identifying  and  discussing 
why  the  analysis  community  has  been  slow  to  embrace  ADS.  These  issues 
can  be  loosely  aggregated  into  two  categories:  quality  and  use. 

In  the  area  of  quality,  the  panels  cited  the  following  issues: 

•  Ambiguity.  Since  many  of  the  members  of  the  analysis  community  are  not 
sure  what  ADS  is,  they  are  not  sure  how  useful  it  is  to  them. 

•  Replicability.  Since  many  examples  of  ADS  feature  HITL,  there  is  concern 
among  some  analysts  about  the  ability  to  replicate  results.  In  particular, 
there  is  concern  about  coping  with  the  effects  of  subject  learning  and  the 
impact  of  differences  in  subject  background,  experience,  proficiency  and 
morale. 

•  Credibility.  It  was  acknowledged  by  several  panels  that  W&A  of  ADS 
pose  particular  challenges. 


Why  Have  Analysts  Not  Embraced  ADS?  (Continued) 


•  Suitability  for  Some  Issues.  Several  panels  noted  that  there  are  classes  of 
analysis  problems  for  which  ADS  may  not  be  suitable.  This  can  stem  from 
the  limited  knowledge  base  that  may  exist  for  a  class  of  problems  (e.g., 
studies  of  new  classes  of  ships  that  are  still  in  the  preliminary,  conceptual 
stage)  or  the  limited  precision  that  one  can  hope  to  achieve  using  some 
classes  of  ADS  (due  to  the  likelihood  of  relatively  small  sample  sizes). 

In  the  area  of  use,  the  panels  cited  the  following  issues: 

•  Cost/Resources  Implications.  Several  panels  obsen/ed  that  it  is  likely  to  be 
relatively  expensive  to  acquire,  update  and  employ  ADS  in  comparison  to 
more  traditional  analysis  tools.  In  addition,  from  a  resource  perspective,  it  is 
often  difficult  to  obtain  and  train  suitable  test  subjects. 

•  Time/Schedule  Implications.  Numerous  panels  expressed  concern  about 
the  extensive  time  implications  of  ADS.  This  includes  the  time  to  create  an 
ADS  capability,  set-up  conditions  for  a  specific  analysis,  perform  test  runs 
and  reduce  the  data  that  is  generated  by  exercising  an  ADS.  These  time 
considerations  can  preclude  use  of  ADS  when  the  analyst  is  given  little  lead 
or  execution  time  (e.g.,  program  objective  memorandum  issue  analyses). 


Why  Have  Analysts  Not  Embraced  ADS?(Conclusion) 

0  Use  Issues  (Conclusion) 

-  Data  implications;  probleihs  associated  with : 

»  Acquiring  data  to  set  Up,  execute  ADS 
i  »  Deciding  what  diata  to  collect  arid  collecting  it 

-  Exercising  the  ADS  (technicaliy) 

-  ■  :-»':ICoritrol'issues  ■  , 

:  »  Time  synchronization 

»  Interconnection  Issues  (e.g.,  immaturity  of  protocols)  :  :  i 

.  .  Security  Issues  ^  ^ 

-  Administrative  burden;  e.g., 

»Cross-commuriity/organlzationcoordination  - 

Scheduling  facilities  ^ 

»  Arranging  for  test  subjects 
»  Configuration  management 

-  Efficient,  effective  exercising  of  ADS;  experimentai  designs  that: 

>  Pro>dde  acceptable  confidence  levels 

:  »  Cope  with  potential  confounding  factors  (e.g.,  learning  by  test  subjects) 
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Why  Have  Analysts  Not  Embraced  ADS?  (Continued) 

•  Data  Implications.  Several  participants  identified  a  broad  spectrum  of  data 
issues  associated  with  ADS.  These  include  problems  associated  with 
acquiring  data  needed  to  set-up  and  execute  ADS;  deciding  what  data  to 
collect  and  collecting  it  (note:  it  may  be  difficult  to  introduce  ad  hoc  “hooks” 
into  an  ADS  to  collect  desired  data);  and  storing  and  processing  the 
resulting  data  (e.g.,  some  of  the  data  of  interest  may  be  spoken  by  the  test 
subjects). 

•  Exercising  the  ADS,  Technicaliy.  Several  ADS  practitioners  identified  the 
problems  that  they  encountered  in  technically  exercising  an  ADS.  To 
illustrate  the  problem  of  control,  a  representative  from  the  Theater  Air  C2 
Simulation  Facility  (TACCSF)  recalled  an  incident  where  a  frustrated 
AWACS  subject  unrealistically  walked  from  his  simulated  node  to  a 
simulated  ground  node  to  resolve  a  concept  of  operations  issue.  Second, 
the  distribution  of  nodes  can  pose  time  synchronization  issues  (e.g.,  the 
use  of  satellite  communications  can  give  rise  to  latencies  that  are 
unacceptable  in  selected  classes  of  analyses  such  as  real  time,  extended 
air  defense  problems).  Third,  several  practitioners  pointed  out  that 
interconnection  of  heterogeneous  nodes  is  still  in  its  infancy.  The  DIS 
standards/protocols  are  still  incomplete  and  evolving. 


Why  Have  Analyst’s  Not  Embraced  ADS?  (Continued) 

In  addition,  the  ambiguity  in  existing  standards/protocols  is  such  that  two 
nodes  can  claim  to  be  DIS  compatible  and  still  not  be  able  to  interoperate. 
Finally,  the  absence  of  multilevel  security  solutions  makes  it  difficult  to  conduct 
experiments  with  ADS  featuring  nodes  at  different  classification  levels. 

•  Administrative  Burdens.  ADS  can  pose  administrative  burdens  that  are  far 
greater  than  those  normally  encountered  by  an  analyst.  For  example,  if  the 
analyst  is  truly  to  leverage  the  resources  of  others  (either  those  of  other 
analysts  or  of  other  communities  such  as  training  or  acquisition)!  it  will  entail 
extensive  cross-community/organization  coordination.  As  an  ancillary  problem, 
it  is  generally  onerous  to  schedule  the  facilities  of  others,  particularly  when  the 
facilities  are  widely  distributed.  In  addition,  for  those  ADS  that  feature  HITL,  it 
can  be  quite  difficult  arranging  for  test  subjects  (i.e.,  getting  people  with  the 
right  experience,  knowledge  and  skills  for  the  times  needed).  Lastly, 
configuration  management  poses  problems  since  it  must  be  maintained  as 
facilities  are  modified  to  meet  analysis  needs  and  to  track  real  world  evolution 
of  systems. 


Why  Have  Analyst’s  Not  Embraced  ADS?  (Continued) 

•  Efficient,  Effective  Exercising  of  ADS.  In  order  to  employ  ADS  efficiently  and 
effectively,  it  is  necessary  to  devise  and  implement  experimental  designs  that 
provide  confidence  levels  consistent  with  the  issues  under  study  and  cope 
with  potential  confounding  factors  that  may  be  unique  to  ADS  (e.g., 
compensate  for  potential  learning  by  test  subjects ). 


Why  Should  Analysts  Bother  With  ADS? 

0  ADS  has  the  potential  to  enhance  substantially  analytic 
support  to 

.  Operations  (e.g.,  dramatically  enhance  quality  of  “what  if...” 
analyses) 

-  Acquisition  (e.;g;  j  revolutionize  the  process  via  simulation  based 
design) 

0  ADS  may  enable  analysts  to  address  issues  |n  a  richer  context : 
than  they  otherwise  could;  e.g.: 

-  Consider  information  aspectsof  warfare 

-  Treat  humaii  Interactions  explicitly 

-  Address  joint  issues  more  credibly 

0  Technology  trends  will  make  ADS 
-More  capable 

-  Cheaper  to  create 

-  Easier  to  use  and  reuse  :  ^ 
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Why  Should  Analysts  Bother  With  ADS? 

In  view  of  the  many  issues  associated  with  ADS,  it  poses  the  question  "Why 
should  analysts  bother  with  ADS?"  The  Workshop  panels  identified  several 
significant  opportunities  implicit  within  ADS  that  suggest  that  it  is  well  worth 
the  time  of  the  analysis  community  to  explore  the  application  of  ADS  to  a 
broad  range  of  analyses. 


First,  several  panels  reported  that  ADS  has  the  potential  to  enhance 
substantially,  analytic  support  to  a  number  of  application  areas.  For 
example,  in  the  area  of  operations,  ADS  has  the  potential  to  enhance 
dramatically  the  quality  of  "what  if..."  analyses.  As  an  illustration,  an 
extension  of  ARPA's  Project  Odin  technology  might  enable  operational 
commanders  to  visualize  the  latest  intelligence  data  and  experiment  with 
alternative  tactics  and  procedures.  In  the  area  of  acquisition,  it  was  noted 
that  the  state-of-the-practice  has  enabled  several  commercial  firms  to 
improve  the  quality  of  their  products  (e.g.,  Boeing's  use  of  CATIA  to  develop 
the  "snap  together"  777).  ARPA  is  at  the  state-of-the-art  level  in  this 
application  and  they  are  optimistic  that  it  can  be  used  to  reduce  the  cost  and 
schedule  of  an  acquisition  while  enhancing  the  quality  of  the  product  (e.g., 
the  simulation  based  design  activities  for  acquiring  future  naval  platforms). 


Why  Should  Analyst’s  Bother  With  ADS?  (Continued) 

Second,  ADS  may  enable  analysts  to  address  issues  in  a  richer  context  than 
they  otherwise  could.  As  illustrations,  note  that  the  use  of  HITL  makes  ADS  an 
attractive  candidate  to  treat  human  interactions  explicitly  and  to  consider 
information  aspects  of  warfare.  In  addition,  the  ability  to  network  the  tools  from 
several  Services  suggests  that  it  should  help  to  address  joint  issues  more 
credibly. 


Third,  technology  trends  are  providing  new  opportunities.  The  Information 
Technology  (IT)  that  underlies  ADS  is  being  enhanced  dramatically  by  the 
commercial  sector  (e.g.,  substantial  improvements  in  processing  speeds, 
storage  size  and  visualization  tools  at  significantly  reduced  costs).  In  addition, 
efforts  are  underway  in  the  defense  community,  such  as  the  HLA,  to  promote 
interoperability  and  reuse.  The  net  effect  of  these  trends  is  to  suggest  that  ADS 
will  become  increasingly  more  capable,  cheaper  to  create  and  easier  to  use 
and  reuse. 


Why  Should  Analysts  Bother  With  ADS?  (Conclusion) 

0  ADS  will  facilitate  leveraging  other  communities’  resources, 
expertise  and  data 

0  ADS  may  enable  us  to  “return  to  our  roots”  (e.g.,  computer 
assisted  analysis  vice  Computer  driven  analysis) 

0  Decision  makers  may  demand  that  analysts  employ  it 

0  It  is  coming  and  it  will  be  pervasive  in  the  defense  community 

(“In  ten  yearSr  everything  can  be  connected  to  everything  eiSe 
...  and  it  will!”  Jack  Thorpe) 

6  ADS  may  be  the  precursor  to  full  “Instrumentation”  of  every 
cdhibatant  on  the  batliefieldj  leading  to  unprecedented  data 
for  understanding  warfare 

Why  Should  Analysts  Bother  With  ADS?  (Continued) 

Fourth,  as  observed  by  Dr.  Jones,  the  anal^ic  community  has  the 
opportunity  to  take  advantage  of  the  extensive  ADS  investments  of  other 
communities  (most  notably  the  training  community).  If  done  properly,  this 
could  provide  the  analysis  community  with  ready  access  to  expertise  (e.g., 
knowledge  on  how  to  use  ADS  effectively  and  efficiently),  resources  (e.g., 
the  opportunity  to  use  existing  ADS)  and  needed  data  and  parameters. 


Fifth,  Mike  Bauman,  TRAC-Ft.  Leavenworth,  and  Gene  Visco,  FS, 
ODUSA(OR),  observed  that  in  recent  years  we  have  seen  the  rise  of 
"computer  driven  analysis."  They  postulated  that  ADS  poses  the  opportunity 
for  analysts  to  move  to  "computer  assisted  analysis"  and  thereby  "return  to 
our  roots." 


Sixth,  it  was  observed  by  several  participants  that  decision  makers  are  being 
exposed  to  ADS  through  its  application  by  other  communities.  Thus,  it  would 
not  be  surprising  if  these  decision  makers  were  to  mandate  the  selective  use 
of  ADS  by  analysts  to  capture  many  of  its  attractive  characteristics  (e.g. 
explicit  treatment  of  human  behavior,  interactive  features  and  high  quality 
visual  features). 
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Why  Should  Analyst’s  Bother  With  ADS?  (Continued) 

Finally,  Jack  Thorpe,  SAIC,  made  two  observations  about  ADS.  He 
predicted  that  “In  ten  years,  everything  can  be  connected  to  everything 
else  and,  given  the  spirit  of  experimentation  that  exists  within  the 
analytic  community,  it  will!"  Thus,  he  opined,  ADS  is  coming  and  it  will 
be  pervasive  in  the  defense  community.  In  addition,  he  speculated  that 
ADS  may  be  the  precursor  to  full  "instrumentation"  of  every  combatant 
on  the  battlefield.  If  that  comes  to  pass,  it  will  lead  to  unprecedented 
access  to  timely  data  for  understanding  warfare.  By  analyzing  these 
data,  analysts  will  have  an  unprecedented  opportunity  to  formulate  and 
validate  models  of  conflict. 


Selected  Findings  and  Recommendations 


0  ADS  Definition 


0  Community  Leadership 
p  ADS  Plan  for  the  Analysis  Comrhunity 


0  Community  Education,  Training  and  Experience 
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Selected  Findings  and  Recommendations 

Based  on  the  insights  that  the  Synthesis  Panel  gained  from  participation  on 
other  panels  and  Synthesis  Panel  discussions,  a  total  of  four  areas  were 
identified  that  the  Panel  members  felt  were  of  overarching  importance.  For 
each  of  these  areas,  the  Synthesis  Panel  developed  findings  and  a  set  of 
recommendations.  In  each  case,  the  Synthesis  Panel  identified  the 
organizations  that  they  believe  should  take  the  lead  in  implementing  the 
recommendations. 


Issue:  ADS  Definition 

0  Findings 

-  There  Is  no  existent  definition  of  ADS  that  was  perceived  as 
adequate  by  the  Workshop  partici|Mints 

•  The  lack  of  an  adequate  definition  limits  the  analytic 
community’s  ability  to 

Understand  the  nature  of  the  capability  (atMbutes/llmitations,  current  or  future) 

»  Plan  for  the  effective  application  of  both  ADS  and  non>ADS  capabilities 

0  Recommendations 

-  A  taxonomy  should  be  developed  that 

»  Defines  and  clarifies  the  nature  of  ADS 
>  Illuminate^  its  relationship  to  related  concepts  (e.g.»  DtS) 

(Action:  AOS  SAG;  DMSO) 
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Issue:  ADS  Definition 


The  Synthesis  Panel  obsen/ed  that  there  is  no  existent  definition  of  ADS  that 
is  deemed  adequate  by  the  Workshop  participants.  The  Panel  found  that 
this  lack  of  an  adequate  definition  limits  the  analytic  community's  ability  to 
understand  the  nature  of  the  capability  (i.e.,  to  appreciate  its  current  or  future 
attributes  or  limitations).  This,  in  turn,  limits  the  analytic  community's  ability 
to  plan  for  the  effective  application  of  both  ADS  and  non-ADS  M&S 
capabilities. 


Consequently,  the  Synthesis  Panel  recommends  that  a  taxonomy  should  be 
developed  that  defines  and  clarifies  the  nature  of  ADS  and  illuminates  its 
relationship  to  related  concepts  such  as  DIS.  The  MORS  ADS  Senior 
Advisory  Group  (SAG)  would  be  an  appropriate  organization  to  draft  this 
taxonomy  and  then  pass  it  on  to  the  DMSO  for  approval  and  dissemination 
throughout  the  community. 


Strawman  Taxonomy  for  ADS 


Factor* 

Alternative  Levels 

Interoperability 

Not 

^^Swivet  Chair” 

Fully  Interoperable 

Distributed 

Not 

Distributed  Data  ... 

Distributed  Data  and 
Processing 

Interactive 

Not 

Sequential 

Highly  Interactive 

Human 

Representation 

Not 

Constructive 

HITL 

Non-ADS 

Variations  of  ADS 

*  other  factors  of  interest  might  include:  level  of  technology  (e.g.,  processing, 
interfaces),  representation  granularity,  architecture  (e.g.,  object-oriented), 
protocois/standards  and  infrastructure 


Strawman  Taxonomy  for  ADS 

As  a  point  of  departure  for  that  task,  a  strawman  taxonomy  for  ADS  is  put 
forth.  The  chart  above  identifies  a  selected  set  of  factors  that  may  prove 
useful  in  discriminating  between  ADS  and  non-ADS  M&S.  The  factors  cited 
include  some  of  the  most  commonly  used  descriptors  of  ADS:  the  ability  to 
interoperate  with  selected  M&S,  the  extent  to  which  it  is  geographically 
distributed;  the  degree  of  interactivity  with  other  M&S  and  the  way  in  which 
humans  are  represented.  In  addition,  the  footnote  to  the  figure  identifies  a 
•number  of  additional  factors  that  might  prove  useful  in  discriminating  ADS 
and  non-ADS  capabilities  (e.g.,  the  level  of  technology  employed  the  level  of 
representation  granularity  or  aggregation;  the  architecture  employed;  the 
protocols/standards  that  are  used  and  the  reliance  on  a  community 
infrastructure  such  as  the  Run  Time  Infrastructure  (RTI)). 


It  should  be  observed  that  the  factors  cited  are  generally  not  binary  in  nature 
(e.g.,  a  system  can  take  on  a  spectrum  of  values  of  interoperability).  Thus, 
the  proposed  taxonomy  should  be  viewed  as  an  array  in  which  selected 
contours  will  can/e  out  communities  that  share  interesting  common 
properties.  As  a  limiting  example,  it  is  suggested  that  M&S  that  share  the 
common  properties  of  being  non-interoperable,  non-distributed;  non¬ 
interactive  and  not  representing  human  behavior,  should  be  excluded  from 
the  ADS  category. _ 


Strawman  Taxonomy  for  ADS  (Continued) 

As  a  side  comment;  several  participants  at  the  Workshop  identified  DIS- 
compatibility  with  ADS.  However,  the  acquisition  community  is  more  prone  to 
employ  tools  that  use  Standards  for  the  ^change  of  Product  Model  Data 
(STEP)  protocols  and  standards.  This  suggests  that  the  factor 
■protocois/standards"  should  be  employed  in  the  taxonomy  and  that  ADS 
should  transcend  the  DIS  limitation. 
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Issue:  Community  Leadership 

^;0 'Findings-' 

.  The  chaileriges  associated  with  ADS  transcend  the  abilities,  resources 
of  the  individual  analyst 

-  If  analyst&are  to  iise  ADS  effectively,  a  paradigm  shift  must  occur  to 
enha/jced  co//aboiaf/ve  ana/ys/s,  built  on  the  “pillars”  of  cross- 
community  shared  data,  security,  Measures  of  Merit  (MoMs)  and  an 
integrated  family  of  advanced  analysis  techniques 

0  Recommendations 

-  Analytic  organizationsfthe  analytic  community  must  stimulate/lead  the 
paradigm  ^Ift  by: 

»  Maidng  nec^sary  investments  in  community  infrastructure  efforts  (e.g., 
consider  estaWishIng  a  “PM  Analytic  ADS”  organization  and  budget  line 

.  ,  :|tems)"V-  ;  ■ 

»  Facilitating  needkl  education,  training 

»  Forming  teams  with  the  nwessary  mix  of  skills  (e.g.,  OR,  experimental  design, 
program  management  and  data  minoig) 

(Action:  Services,  Joint  staff,  OSD) 


Issue:  Community  Leadership 

The  Synthesis  Panel  found  that  the  challenges  associated  with  ADS 
transcend  the  abilities  and  resources  of  the  individual  analyst.  If  analysts  are 
to  use  ADS  effectively,  a  paradigm  shift  must  occur  to  enhanced 
collaborative  analysis.  This  concept,  which  was  put  forth  by  the  Analysis  and 
Requirements  Panel,  rests  on  pillars  of  cross-community  shared  data,  tools 
(e.g.,  scenario  generation  tools,  visualization  tools),  security.  Measures  of 
Merit  (MoMs)  and  an  integrated  family  of  analysis  techniques.  The  last  of 
these  pillars  should  be  emphasized.  Most  of  the  panels  observed  that  ADS 
should  not  be  viewed  as  a  "stand-alone"  capability  that  can  be  applied  to  all; 
analytic  problems.  In  general,  it  must  be  harmonized  and  orchestrated  with 
other  analysis  techniques  to  compensate  for  selected  ADS  features  (e.g., 
extensive  time  to  create  and  execute)  and  employed  where  appropriate. 

In  order  to  bring  about  this  paradigm  shift,  key  analytic  organizations  and  the 
analytic  community  must  provide  needed  stimulation  and  leadership.  A 
partial  listing  of  recommended  actions  to  be  undertaken  by  the  Services,  the 
Joint  Staff  and  OSD  include  the  following: 


•  Providing  incentives  to  use  ADS.  This  would  include  providing 
appropriate  resources  to  analysts  and  promoting  those  analysts  who 
employ  ADS  appropriately  and  effectively. 


Issue:  Community  Leadership  (Continued) 

•  Making  necessary  investments  in  community  infrastructure  and 
accreditation  efforts.  It  was  observed  that  the  analysis  community  is 
normally  limited  to  "chump  change."  To  provide  the  resources  needed  to 
develop  and  VV&A  ADS,  several  members  of  the  Synthesis  Panel 
recommended  that  a  "PM  Analytic  ADS"  organization  be  estabjished  and 
supported  with  adequate  budget  line  items.  This  recommendation  should 
be  assessed  further  to  establish  its  feasibility  and  desirability. 

•  Facilitating  needed  education,  training.  The  cornerstone  to  effective 
creation  and  use  of  ADS  is  the  education  and  training  of  the  analyst  and 
the  decision  maker.  In  view  of  the  criticality  of  this  step,  it  is  amplified 
upon  as  a  separate  issue  below. 

•  Forming  teams  with  the  necessary  mix  of  skills.  The  effective  creation 

and  use  of  ADS  demands  a  team  effort.  Representative  skills  include  OR 
(to  help  frame  and  scope  the  problem),  experimental  design  (to  formulate 
the  experimentat  design  matrices  and  support  the  analysis  process), 
program  management  (to  perform  the  myriad  management  and 
administrative  tasks)  and  data  mining  (to  acquire,  process  and  store  data 
associated  with  ADS).  _ _ 


Issue:  ADS  Plan  for  the  Analysis  Community 

0  Observations 

~  Historically,  spokespersons  for  the  analysis  community  had  resisted 

»  Identifying  Its  needs  for  acWancedM&S  On  comrast  to  other  conanunW^ 

»  Developing  a  proactive  plan  to  redress  M&S  r^dencies 

-  There  is  no  long  range  vision  to  guide  the  allocation  of  limited  M&S 
development  resources  to  support  the  analyst  —we  tend  to  be  reactive  , 
to  near-^term  needs 

0  Findings 

-  The  absence  of  an  “ADS  Plan  for  the  Analysis  Community”  has  resisted 
in 

»  Ragmented  action  witMh  the  <x>m^ 

»  Limited  leveraging  of  ct’oss'coimnunlty  resources,  expertise  . 

0  Recommendations 

-  The  Analysis  Council  should  take  on  the  responsibility  of 

.  :  »  Developing  the  vision  for  the  next  generation  of  M&S  tools  for  the  analyst 

»  Drafting  “ADS  Plan  for  the  Airalysis  Community  ”  ho  later  than  1  February  1997 

-  The  plan  should  explicitly  identify  steps  to  ensure  maximum 

(Action:  Analysis  Council) 

. 
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Issue:  ADS  Plan  for  the  Analysis  Community 

Once  the  analysis  community  becomes  committed  to  the  enhanced 
collaborative  analysis  paradigm,  it  will  need  a  plan  to  guide  its  actions. 
Historically,  the  analysis  community  has  been  reluctant  to  take  this  step.  For 
example,  several  years  ago,  DMSO  sponsored  a  series  of  workshops  to 
stimulate  the  various  communities  to  identify  their  needs  for  advanced  M&S. 
While  several  communities  embraced  this  opportunity  (e.g.,  training,  T&E), 
the  analysis  community  demurred.  The  Synthesis  Panel  further  observed 
that  there  is  no  long  range  vision  to  guide  the  allocation  of  limited  M&S 
development  resources  to  support  the  analyst.  Consequently,  the  analysis 
community  tends  to  be  excessively  reactive  to  near-term  needs  at  the 
expense  of  needed  longer-term  investments. 


The  Synthesis  Panel  found  that  the  absence  of  an  "ADS  Plan  for  the 
Analysis  Community"  has  resulted  in  fragmented  action  within  the  analysis 
community  and  limited  the  leveraging  of  the  resources  and  expertise  that  are 
resident  in  other  communities  (e.g.,  training,  acquisition).  To  ameliorate 
those  shortfalls,  the  Analysis  Council  that  has  been  formed  under  the 
Executive  Committee  for  M&S  (EXCIMS),  must  take  decisive  action.  It 
should  develop  the  vision  for  the  next  generation  of  M&S  tools  for  the 
analyst  and  draft  an  associated  "ADS  Plan  for  the  Analysis  Community." 


Issue:  ADS  Plan  for  the  Analysis  Community  (Continued) 

In  the  view  of  the  need  for  timely  action,  these  products  should  be  drafted 
£ind  circulated  for  community  coordination  no  later  than  1  February  1997. 
To  ensure.that  the  community  takes  full  advantage  of  the  opportunities 
implicit  in  commercial  developments,  the  plan  should  explicitly  identify 
steps  to  ensure  maximum  use  and  compatibility  between  Commercial-Off- 
The-Shelf  (COTS)  products  and  ADS. 


Issue:  Community  Education,  Training  and  Experience 

0  Findings 

-  A  broad  of  skills,  knowledge,  and  expertise  will  be  needed  by  the 

community  if  it  is  to  be  able  to  employ  ADS  effectively  and  efficiently 

-  In  the  absence  of  appropriate  education,  training,  and  experience, 

»  The  resutts  that  emerge  from  the  application  of  ADS  are  likely  to  be  suspect 
»  Dedsion  maker  understandlng  of  the  meaning^  validity  of  results  is  likely  to  be  limited 

0  Recommendations  ^  ^  ^  ^  ^  ^ 

-  Key  Service  schools  should  develop  curricula  and  offer  a  sub*speclaity  of 
“ADS  Analysf-  (Action:  NFS,  AFIT  and  ALMC) 

-  Assignments  to  '^DS  Anaiy^”  positions  should  be  of  suitable  duration  to 
ensure  that  necessary  experience  is  acquired  and  sustained 

-  Training  programs  lor  deicisibn  makers  should  be  augmented  with  material  '' 

on  ADS  (Action:  DSMC,DAU^  NDU  and  Senior  Service  Schools) 

Issue:  Community  Education,  Training  and  Experience 

The  Synthesis  Panel  found  that  a  broad  set  of  skills,  knowledge  and 
expertise  will  be  needed  ty  the  community  if  it  is  to  able  to  appreciate  the 
capabilities  and  limitations  of  ADS  and  to  employ  ADS  effectively  and 
efficiently.  It  must  be  emphasized  that  the  community  in  question  subsumes 
both  analysts  and  decision  makers.  In  the  absence  of  appropriate  education, 
training  and  experience,  the  results  that  analysts  derive  from  the  application 
of  ADS  are  likely  to  be  suspect  and  the  decision  makers'  understanding  of 
the  meaning  and  validity  of  those  results  is  likely  to  be  limited. 


To  deal  with  these  issues,  the  Synthesis  Panel  put  forth  several 
recommendations.  First,  key  Service  schools  should  develop  curricula  and 
offer  a  sub-specialty  of  "ADS  Analyst."  Candidate  schools  include  the  Naval 
Postgraduate  School  (NPS),  the  Air  Force  Institute  of  Technology  (AFIT)  and 
the  Army  Logistics  Management  College  (ALMC).  Second,  assignments  to 
"ADS  Analyst"  positions  should  be  of  suitable  duration  and  consistency  to 
ensure  that  necessary  experience  is  acquired  and  sustained.  Thus,  for 
example,  once  an  analyst  has  qualified  as  an  "ADS  Analyst"  at  one  of  the 
above  institutions,  his/her  next  assignment  should  be  to  an  ADS  activity  or  to 
an  organization  which  may  perform  collaborative  analysis  employing  ADS. 
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Issue:  Community  Education,  Training  and  Experience  (Continued) 


Note  that  some  Services  currently  assign  recent  graduates  to  field 
assignments.  Subsequently,  a  sequence  of  assignments  should  be  envisioned 
that  help  the  ADS  Analyst  to  continue  to  hone  and  apply  his/her  skills. 

In  support  of  decision  maker  education  and  training,  appropriate  programs  on 
ADS  should  be  developed  and  offered  at  institutions  such  as  the  Defense 
Systems  Management  College  (DSMC),  the  Defense  Acquisition  University 
(DAU),  the  National  Defense  University  (NDU)  and  the  Senior  Service  Defense 
Schools.  Thought  should  be  given  to  updating  DSMC's  Handbook  on  M&S  for 
the  acquisition  community  to  highlight  the  challenges  and  opportunities 
associated  with  ADS. 


Summary  Assessment:  Achievements  Versus  Objectives 


0  Assess  the  utility  of  current  and  future  (3-5  years  out)  ADS- 
based  capabilities  for  analytic  applicafidhs,  especially 
possible  areas  of  new  analytic  capabilities— -  AMBER 

0  Define  areas  of  needed  enhancements  of  AD&based 
capabilities  to  increase  their  utinty  to  the  militaiy  analytic 
community  —  AMBER 

0  Increase  the  military  analytic  coihmuriity’s  understanding  of 
current  ADS-based  capabilities  and  near-term  improvement 
programs  —  GREEN 


increase  interactions  between  the  military  analytic  community 
and  developers  of  ADS-based  capabilities —  GREEN 
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Summary  Assessment:  Achievements  Versus  Objectives 

The  TOR  identified  four  objectives  for  the  Workshop.  T o  establish  a 
qualitative  measure  of  the  effectiveness  of  the  Workshop,  the  Synthesis 
Panel  informally  assessed  the  other  panels  against  these  four  objectives, 
based  on  its  observations  of  their  operations  and  products.  These 
assessments  were  in  the  form  of  a  color  (i.e.,  "green"  Implied  fairly  complete 
achievement  of  the  objective,  "amber"  implied  moderate  achievement  of  the 
objective  and  "red”  implied  a  failure  to  achieve  the  objective). 


The  first  objective  was  to  assess  the  utility  of  current  and  future  (3-5  years 
out)  ADS-based  capabilities  for  analytic  applications,  especially  possible 
areas  of  new  analytic  capabilities.  The  Synthesis  Panel  rated  this  as 
"amber."  There  was  extensive  consideration  of  the  current  time  frame  but 
less  about  the  future  because  of  the  limited  understanding  about  the  likely 
nature  of  ADS  by  the  turn  of  the  century. 


The  second  objective  was  to  define  areas  of  needed  enhancements  of  ADS- 
based  capabilities  to  increase  their  utility  to  the  military  analytic  community. 
The  Synthesis  Panel  rated  this  as  "amber."  It  was  felt  that  the  absence  of  an 
agreed  definition  for  ADS  limited  this  discussion,  although  a  number  of 


Summary  Assessment:  Achievements  Versus  Objectives  (Continued) 

panels  put  forth  excellent  suggestions  for  increasing  the  utility  of  ADS  to  the 
military  analytic  community. 

The  third  objective  was  to  increase  the  military  analytic  community's 
understanding  of  current  ADS-based  capabilities  and  near-term 
improvement  programs.  The  Synthesis  Panel  rated  this  as  "green."  This 
was  due  to  the  inclusion  of  descriptive  material  in  the  read-ahead  package, 
the  material  covered  in  selected  plenary  presentations,  the  scheduling  of 
ADS  presentations  within  selected  panels  and  the  ad  hoc  scheduling  of  a 
presentation  on  the  HI_A  to  the  workshop. 


The  fourth  objective  was  to  increase  interactions  between  the  military 
analytic  community  and  developers  of  ADS-based  capabilities.  The 
Synthesis  Panel  rated  this  as  “green."  This  was  due  to  the  mix  of 
participants  that  were  invited  to  the  Workshop  and  the  opportunities  that 
were  provided  to  enable  them  to  have  a  fruitful  dialogue. 
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TERMS  OF  REFERENCE 
MORS  WORKSHOP: 

AD  VANCED  DISTRIBUTED  SIMULA  TIONS for 
ANALYSIS  (ADSA  ‘96) 

November  27, 1995 

1 .  Goal:  The  primary  goal  of  this  workshop  is  to  assess  the  applicability  of  current  and  near- 
term  Advanced  Distributed  Simulation  (ADS)  capabilities  as  applied  to  military  analysis.  A 
secondary  goal  of  this  workshop  is  to  stimulate  communication  regarding  the  continuing  evolution 
of  Advanced  Distributed  Simulation  capabilities,  and  to  foster  interaction  between  the  military 
analytic  community  and  developers  of  ADS-based  capabilities. 

2.  Background:  In  September  of  1992  MORS  sponsored  an  ADS  mini-symposium  with  the 
objective  of  familiarizing  the  military  operations  research  community  with  the  concept  and 
capabilities  of  ADS  and  synthetic  combat  environments.  A  second  goal  was  to  have  participants 
explore  the  use  of  these  capabilities  to  meet  operations  research  and  analysis  needs.  The 
objectives  of  that  conference  were  accomplished  and  a  Proceedings  has  been  published. 

Over  the  last  three  years  a  considerable  amount  of  work  has  been  initiated  across  the 
Department  of  Defense  ^oD)  with  respect  to  the  use  and  development  of  ADS  capabilities. 
Major  technology  demonstrations  have  been  conducted  and  many  programs  have  been  initiated 
within  the  department  to  continue  development  of  ADS  capabilities.  In  March  1994,  MORS 
organized  a  Senior  Advisory  Group  (SAG)  to  review  and  coordinate  Society  activities  related  to 
ADS.  As  a  result  of  the  continued  emphasis  and  importance  that  the  DoD  has  placed  on  the  use 
of  modeling  and  simulation,  the  ADS  SAG  recommended  that  MORS  convene  a  workshop  to 
pursue  a  more  detailed  examination  of  the  use  of  ADS  by  the  mihtary  operations  research 
community. 

Numerous  reports  identify  the  fact  that  our  military  forces  now  face  an  extensive  variety 
of  potential  adversaries,  presenting  US  military  forces  Avith  a  wide  spectrum  of  missions  and 
operations,  including  “other  than  war”  roles.  Many  of  these  stress  aspects  of  the  use  of  military 
forces  beyond  that  of  conventional  analysis  of  “attrition  combat.”  The  DoD  continues  to  stress 
the  need  for,  and  potential  of,  application  of  advanced  modeling  and  simulation  capabilities  to 
increase  the  readiness  of  our  military  to  execute  the  roles  and  missions  necessary  to  achieve  the 
objectives  of  our  National  Military  Strategy. 

Many  changes  have  occurred  with  regard  to  ADS  over  the  past  two  years.  For  example, 
all  of  the  military  services  now  have  plans  for  new  constructive  models  utilizing  many  ADS-based 
technologies;  all  of  the  services’  major  current  constructive  models  now  interact  with  current 
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DIS-based  virtual  systems;  ARPA  has  a  new  major  program  underway  to  expand  the  capabilities 
of  synthetic  theater  of  war  environments  (STOW-97).  DMSO  has  recently  promulgated  a  draft  of 
a  DoD-wide  high  level  architecture  for  modeling  and  simulation;  and  a  master  plan  for  achieving 
ADS-based  capabilities  and  improving  modeling  and  simulation  capabilities  overall.  Additionally, 
all  of  the  Services  have  a  considerable  number  of  initiatives  underway  to  improve  and  to  integrate 
many  live,  constructive  and  virtual  instrumentation,  and  modeling  and  emulation  capabilities. 

Also,  recently  a  Joint  Simulation  (JSIMs)  Program  Management  Office  has  been  created  to 
oversee  development  of  joint  simulations.  Recently  the  Theater  Missile  Defense  (TMD) 
community  has  recognized  the  potential  of  ADS  to  facilitate  testing,  exercise,  and  demonstration 
of  its  systems  in  the  environment  of  wartime  operations  that  cannot  be  created  in  any  practical 
way  using  real  threat  systems.  DoD  needs  such  as  these  represent  an  analytical  challenge  that 
must  be  addressed  by  the  militaiy  operations  research  community  as  well  as  an  operational 
challenge  faced  by  the  war  fighters.  This  major  evolution  of  modeling  and  simulation  capabilities 
requires  that  the  military  analytic  community  understand  both  the  uses  and  limitations  of  ADS- 
based  capabilities,  current  and  potential;  and  that  they  participate  in  defining  the  needs  of  the 
analj^ic  community  with  regard  to  ADS-based  capabilities. 

3.  Objectives: 

a.  The  purpose  of  this  workshop  is  to  achieve  the  following  objectives: 

(1)  Assess  the  utility  of  current  and  future  (3-5  years  out)  ADS-based 
capabilities  for  analytic  applications,  especially  possible  areas  of  new  analytic  capabilities. 

(2)  Define  areas  of  needed  enhancements  of  ADS-based  capabilities  to  increase 
their  utility  to  the  military  analytic  community. 

(3)  Increase  the  military  analytic  community’s  understanding  of  current  ADS- 
based  capabilities  and  near-term  improvement  programs. 

(4)  Increase  interactions  between  the  military  analytic  community  and 
developers  of  ADS-based  capabilities. 

b.  Within  these  broad  objectives  the  workshop  will  focus  on  the  foUoAving  subject 

areas; 

(1)  Analysis  associated  with  determining  the  battlefield  effectiveness  of  operational 
concepts,  tactics,  doctrine  and  forces. 

(2)  Analysis  associated  with  determining  material  requirements,  system  to 
system  comparisons  of  battlefield  effectiveness,  system  life  cycle  design  tradeoffs,  and  decisions 
pertaining  tc  the  acquisition  process. 
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(3)  Analysis  associated  with  Test  and  Evaluation. 

(4)  Analysis  associated  with  determining  the  effectiveness  of  training,  mission 
rehearsal,  and  alternate  courses  of  action  during  operational  planmng. 

(5)  Analysis  and  requirements,  including  processes  for  assessing  and  improving 
ADS-based  analytic  tools. 

(6)  Analysis  of  DIS  simulations  with  the  objective  of  deepening  understanding 
of  warrior  behavior/performance  and  then  incorporating  that  understanding  in  improved 
constructive  models  and  data. 

c.  A  working  group  will  be  convened  to  address  each  of  the  above  subject  areas. 

Each  working  group  will  be  specifically  charged  to  address  broad  analytic  questions  "within  its 
analytic  domain  to  include: 

•  When  should  ADS-based  capabilities  be  used?  (For  what  types  of  analysis  and  to 
answer  what  kinds  of  analytic  questions?) 

•  Are  ADS-based  capabilities  better  or  worse  for  specific  types  of  analysis  already 
being  done?  (Which,  how,  why,  when  in  the  analytic  process?) 

•  Do  ADS-based  capabilities  allow  analysis  to  address  any  new,  necessary  analytic 
questions? 

•  How  do  ADS-based  capabilities  fit  with  other  types  of  analytic  capabilities  and 
approaches? 

•  What  are  limitations  of  current  and  future  ADS-based  capabilities?  (Which  are 
generic  limitations  and  which  are  limitations  which  could  be  mitigated  or 
eliminated  with  enhancements  to  ADS-based  capabilities?) 

•  Are  there  pitfalls  in  analysis  using  ADS-based  capabilities?  (What  are  they,  are 
they  generic?) 

•  Does  the  use  of  ADS-based  capabilities  affect  the  utility  of  analytic  results?  (How, 
why,  when?) 

•  Are  there  useful  ways  to  utilize  the  results  fi"om  ADS-based  analysis,  exercise,  and 
training  that  differ  fi-om  conventional  methods?  (How,  when,  under  what 
conditions  and  assumptions?) 
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•  Do  ADS-based  analysis  capabilities  make  verification,  validation  and  accreditation 
processes  any  harder,  easier,  or  different?  If  so,  in  what  ways?  What  are  suggested 
ways  to  take  advantage  of  these  differences  or  to  mitigate  them? 

In  addressing  these  questions,  each  working  group  will  consider  them  in  the  context  of 
scenario  development,  robustness  of  analytic  results,  consideration  of  a  variety  of  adversaries  and 
missions,  comparisons  between  analytic  results  and  field  experience,  other  issues  affecting  the 
utility  of  analysis,  and  other  specific  issues  raised  by  the  workshop  participants. 

4.  Approach: 

a.  The  working  groups  will  consist  of  a  strictly  limited  number  and  mixture  of 
analysts  interested  in  ADS  applications,  persons  with  hands-on  experience  using  existing  ADS 
capabilities,  and  developers  of  advanced  ADS  capabilities.  Working  group  size  will  be  limited  to 
15  persons  to  ensure  adequate  opportunity  for  participation  in  the  discussion.  Six  working 
groups  will  specifically  address  the  areas  delineated  in  Section  3b.  In  addition,  working  groups 
may  also  address  applications  of  ADS  capabilities  to  analysis  of  the  BMD  system. 

b.  A  read  ahead  package  will  be  sent  to  all  participants  to  provide  background 
information  and  an  overview  of  key  ADS  capabilities  and  objectives.  This  will  include  a  baseline 
description  of  service  analytical  needs,  widely  used  models,  simulations,  tools,  and  techniques 
along  with  the  recognized  shortfalls  of  these  capabilities  identified  to  date  by  both  the  ADS  user 
and  development  communities. 

c.  The  first  day  of  the  workshop  will  be  primarily  devoted  to  information  briefings  to 
ensure  that  participants  have  a  broad  understanding  of  current  ADS-based  capabilities,  on-going 
development  programs,  and  key  concepts  and  plans  as  they  relate  to  analysis.  The  latter  portion 
of  the  afternoon  will  be  devoted  to  the  initial  meetings  of  working  groups,  their  organization,  and 
the  introduction  of  participants  to  one  another.  The  first  day  will  include  an  evening  social  to 
facilitate  the  interaction  of  participants.  The  second  day  and  morning  of  the  third  day  will  be 
devoted  to  individual  working  group  sessions.  The  early  afternoon  of  the  third  day  vrill  be 
devoted  to  short  presentations  by  each  working  group  and  a  summary  of  the  workshop.  The 
ADS  SAG  will  meet  in  the  late  afternoon  after  the  workshop  participants  have  been  released. 

d.  Synthesis  Group:  A  small  synthesis  group  will  be  formed  to  observe  the  activities 
and  discussion  of  the  working  groups.  This  group  will  assist  in  identifying  important  and  cross¬ 
cutting  subjects  among  the  groups. 

5.  Membership:  Representation  will  include  DoD,  industry  and  academia.  Working  group 
chairs  will  be  subject  matter  experts  in  their  session  area.  Membership  in  the  working  groups  will 
be  determined  by  the  working  group  chairs,  and  will  be  lunited  in  number  to  facilitate  the 
preparation  of  focused  products. 
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6.  Products:  MORS  workshops  are  designed  to  examine  problems  related  to  military 
operational  analyses  and  to  identify  approaches  to  solutions  of  those  problems.  Products  of 
MORS  workshops  are  (1)  clarification  of  problems  and  problem  areas  and  (2)  identification  of 
solution  “spaces.”  The  clarification  of  problems  are  summarized  in  terms  of  observations  and 
findings  of  the  workshops.  Identification  of  solution  “spaces”  result  in  recommendations  for 
action  by  the  MORS  community  (Department  of  Defense  institutions,  other  government  elements, 
private  sector  organizations,  and  academic  institutions).  Of  special  interest  are  recommendations 
particularly  directed  to  the  MORS  sponsoring  agencies.  The  products  are  conveyed  by:  (1) 
briefings  to  the  workshop  proponents  and  other  sponsors  as  requested  (within  30  days  of  the  end 
of  the  workshop);  (2)  proceedings  consisting  of  an  executive  summary  (including  key  findings  and 
recommendations,  and  common  and  most  prevalent  themes),  a  report  fi'om  each  working  group  at 
the  workshop  with  findings  and  recommendations,  and  papers  and  briefings  presented 
(proceedings  are  targeted  for  production  within  three  months  of  the  end  of  the  workshop);  and 
(3)  an  article  for  the  earliest  issue  of  PHALANX  following  the  end  of  the  workshop. 

The  products  of  this  ADS  workshop  will  include: 

•  Specification  of  classes  of  analyses  most  amenable  to  support  by  ADS  and  those 
for  which  ADS  is  not  useful; 

•  Assessment  of  the  “value-added”  of  ADS  for  analyses  presently  underway; 

•  Identification  of  experiments  which  can  be  designed  to  assist  in  the  development  of 
MOEs  for  analysis  to  include  human-related  functions;  and 

•  Ways  to  use  results  fi'om  ADS-based  analyses  that  differ  from  those  of  other 
analyses. 

Recommendations  relevant  to  findings  and  observations  will  be  presented  to  specific 
sponsors,  as  necessary.  Overall,  the  ADS  workshop  will  form  the  basis  of  a  status  report  on  the 
present  and  possible  contributions  to  analysis  afforded  by  ADS,  and  recommendations  for  action 
that  will  increase  the  likelihood  that  ADS  will  make  significant  and  valuable  contributions  to 
military  analysis  in  the  future. 

7.  Proponents:  The  ODUSA(OR),  OCNO  (N81),  HQ  USAF/XOM,  Joint  Stafi^  J8,  OSD 
(PA&E)  have  agreed  to  be  proponents  for  the  workshop.  MCCDC  has  expressed  supportive 
interest. 

8.  Administrative: 

Title:  “Advanced  Distributed  Simulations  for  Analysis  (ADS A  ‘96)” 

Dates:  30,  31  January,  1  February  1996 

Fee:  $150  (Federal  Government) 

$300  (Others) 

Attendance:  Limited  to  115-140  attendees. 

Classification:  Unclassified 
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E-mail :  post j rd@aol . com 


DR  Henry  C  Dnbin 

US  Army  Operational  Test  &  Eval  Comd 
Attn:  CSTE-ZT 

4501  Ford  Avenue 
Alexandria  VA  22302-1458 
OFF  TEL:  (703) -681-9367  DSN:  761-2367 
FAX:  (703) -681-3779 
E-mail :  dubin@optec.army.mil 


Robert  W  Eberth 

Palmer-Eberth 

Suite  115-212 

1350  Beverly  Road 

McLean  VA  22101 

OFF  TEL:  (703 ) -769-3325 

FAX:  (703) -685-9849 

E-mail :  eberthr@smtp-gw.spawar.navy.mil 


Christine  A  Fossett 
US  GAO 

Office  of  Policy,  Room  6800 

441  G  Street,  NW 

Washington  DC  20548 

OFF  TEL:  (202 ) -512-2956 

FAX:  (202) -512-4844 

E-mail:  cfossett®msis . dmso.rail 


DR  John  A  Friel 
RAND 

1700  Main  Street 
PO  Box  2138 

Santa  Monica  CA  90407-2138 
OFF  TEL:  (310) -393-0411 
Ext:  6712 

FAX:  (310) -393-4818 
E-mail :  john__friel@rand.org 


Gary  E  Engel 
McDonnell  Douglas 
MC  -  064-2233 
PO  Box  516 

St.  Louis  MO  63166-0516 
OFF  TEL:  (314) -233-1316 
FAX:  (314) -233-5125 
E-mail:  gengel@gwsmtpol.mdc.com 


W.  Dean  Free 

Chief  of  Naval  Operations  (N812D) 

2000  Navy  Pentagon 

Washington  DC  20350-2000 

OFF  TEL:  (703) -697-3642  DSN:  227-3642 

FAX:  (703) -693-9760 

E-mail :  cno-n812d3@cno.navy.mil 


DR  Randy  Garrett 

Advanced  Research  Projects  Agency 
ARPA/ISO 

3701  N.  Fairfax  Drive 
Arlington  VA  22203-1714 
OFF  TEL:  (703) -696-2373 
FAX:  (703) -696-2201 
E-mail :  rlg®arpa . mil 
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PARTICIPANTS  IN  ADVACNCED  DISTRIBUTED  SIMULATION  FOR  ANALYSIS  1996  WORKSHOP  as  of  02/10/96 
(Cont . ) 


Jeffrey  W  Gerlitz 
COMOPTEVFOR 
7970  Diven  Street 
Norfolk  VA  23505-1498 

OFF  TEL:  (804 ) -444-5085  DSN:  564-5085 
Ext:  3295 

FAX:  (804) -444-3820 
E-mail :  nsapcotf @nosc . mil 


COL  Kenneth  W  Getty 

US  Special  Operations  Command 

JSOFI 

P.O.  Box  71929 
Fort  Bragg  NC  28307-1929 
OFF  TEL:  (910) -432-2865 
FAX:  (910) -432-5467 
E-mail :  gettyk@usasoc . soc . com 


COL  Richard  B  Gibson 
HQ  USAF,  AFSAA/SAS 

Air  Force  Studies  &  Analyses  Agency 

1570  Air  Force  Pentagon 

Washington  DC  20330-1570 

OFF  TEL:  (703) -695-2039  DSN:  225-2039 

FAX:  (703) -614-2455 

E-mail :  rgibson@sunl9 . hq . af . mil 


Michael  D  Gray 
BDM  Engineering  Services  Co 
1801  Randolph  Rd  SE 
Albuquerque  NM  87106 

OFF  TEL:  (505) -846-4139  DSN:  246-4139 
FAX:  (505) -846-1872 


Priscilla  A  Glasow 
SAIC 

Suite  510 

1901  N.  Beauregard  Street 
Alexandria  VA  22311 
OFF  TEL:  (703 ) -824-3412 
FAX:  (703) -379-3778 
E-mail :  pglasow@msis.dmso.mil 


COL  Paul  Hanover 

Marine  Corps  Modeling  &  Simulation  Off. 
MCMSMO  Code  C46MS 
2006  Hawkins  Avenue 
Quantico  VA  22134-5069 

OFF  TEL:  (703 )- 784-2588  DSN:  278-2588 
FAX:  (703) -784-2591 

E-mail :  hanoverp@mqg-smtp3  .  usmc.mil 


Robert  G  Hart ling 

Chief  of  Naval  Operations  (N812D) 

2000  Navy  Pentagon 
Room  4A522 

Washington  DC  20350-2000 

OFF  TEL:  (703) -695-3797  DSN:  225-3797 

FAX:  (703) -693-9760 

E-mail :  rhart@dmso . dtic . dla . mil 


Frederick  E  Hartman 
Foxhall  Group 
1620  Foxhall  Road 
Washington  DC  20007 
OFF  TEL:  (202 ) -298- 7166 
FAX:  (202) -338-4279 
E-mail :  hartman@aol . com 
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PARTICIPANTS  IN  ADVACNCED  DISTRIBUTED  SIMULATION  FOR  ANALYSIS  199$  WORKSHOP  as  of  02/10/9 

(Cont . ) 


DR  Richard  E  Hayes 
Evidence  Based  Research,  Inc 
Suite  330 

1595  Spring  Hill  Road 
Vienna  VA  22182 
OFF  TEL:  (703) -893-6800 
Ext:  26 

FAX:  (703) -821-7742 
E-mail:  ebrinc@aol.com 


Richard  E  Helmuth 
SAIC 

8301  Greensboro  Dr.,  Suite  460 
POB  50132 

McLean  VA  22102-8932 

OFF  TEL:  (703 ) -749-5130 

FAX:  (703) -847-6406 

E-mail :  dick_helmuth@cpqm.saic .  com 


DR  Jacqueline  R.  Henningsen 
OSD  PA&E 

Regional  Assessment  and  Modeling  Div 

1800  Defense  Pentagon,  Room  2C270 

Washington  DC  20301-1800 

OFF  TEL:  (703 ) -697- 0584  DSN:  227-0564 

FAX:  (703) -695-7988 

E-mail:  jhenning@msis.dmso.mil 


Susan  M  Iwanski 

Northrop  Grumman  Corporation 

Advanced  Technology  &  Development  Ctr 

MS  C63-05 

Bethpage  NY  11714 

OFF  TEL:  (516 ) -346-9138 

FAX:  (516) -346-9740 

E-mail :  iwanski®grumman . com 


Heide  E  Heidepriem 

Johns  Hopkins  University/APL 

Room  13-N422 

Johns  Hopkins  Road 

Laurel  MD  20723-6099 

OFF  TEL:  (301) -953-5177 

FAX:  (301) -953-5910 

E-mail :  heide.heidepriem@jhuapl.edu 


DR  Dale  B  Henderson 
Los  Alamos  National  Laboratory 
A  Division 
MS  F606 

Los  Alamos  NM  87545 
OFF  TEL:  (505) -665-2151 
FAX:  (505) -665-5249 
E-mail :  dbh@lanl . gov 


CAPT  James  W  Hollenbach 
DMSO 

Suite  504 

1901  N.  Beauregard  Street 
Alexandria  VA  22311 
OFF  TEL:  (703 ) -998- 0660 
FAX:  (703) -998-0667 
E-mail :  jwh@dmso.mil 


Raymond  E  Jablonski 

US  Army  Edgewood  RD&E  Center 

Attn :  SCBRD-RTM 

Aberdeen  Proving  Ground  MD  21010-5423 
OFF  TEL:  (410) -671-3566  DSN:  584-3566 
FAX:  (410) -671-3523 

E-mail :  re j  ablon®cbdcom . apgea . army . mil 
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PARTICIPANTS  IN  ADVACNCED  DISTRIBUTED  SIMULATION  FOR  ANALYSIS  1996  WORKSHOP  as  of  02/10/96 
(Cont . ) 


LtCol  Jack  A  Jackson 
AFIT/ENS 

2950  P  Street,  Bldg  640 

Wright -Patterson  AFB  OH  45433-7765 

OFF  TEL:  (513) -255-2579  DSN:  785-6565 

FAX:  (513) -476-4943 

E-mail :  jjackson@afit.af.mil 


Hon  Anita  K  Jones 

Director  Defense  Research  &  Engineering 
Washington  DC  20301 

OFF  TEL:  (703 ) -697-5776  DSN:  227-5776 
E-mail:  Russ  Herndon,  Special  Assistant 


MAJ  Carol  Lynn  Judge 
HQ  USAF/AFSAA/SAGH 

Air  Force  Studies  &  Analyses  Agency 

1570  Air  Force  Pentagon 

Washington  DC  20330-1570 

OFF  TEL:  (703 ) -695-5283  DSN:  225-5283 

FAX:  (703) -697-1226 

E-mail :  judge®af saa . hq . af . mil 


James  C  Keith 
USASSDC 
CSSD-BC-SS 
PO  Box  1500 

Huntsville  AL  35807-3801 
OFF  TEL:  (205) -955-3580 
FAX:  (205) -955-5136 

E-mail :  kei th j ©ssdch-usassdc . army . mil 


DR  Jerry  A  Kotchka 
McDonnell  Douglas  Aerospace 
Mail  Code  0641251 
PO  Box  516 

St.  Louis  MO  63166-0516 
OFF  TEL:  (314) -232-2284 
FAX:  (314) -232-7917 


Eric  L.  Keck 

Joint  Advanced  Distributed  Simulation 
Joint  T&E 
1951  2d  Street,  SE 
Kirtland  AFB  NM  87117-5617 
OFF  TEL:  (505) -846-0580  DSN;  246-0580 
FAX:  (505) -846-0603 
E-mail :  keck@jads.kirtland.af.mil 


DR  Robert  M  Kerchner 
RAND 

2100  M  Street,  NW 
Suite  800 

Washington  DC  20037-1270 
OFF  TEL:  (202 ) -296-5000 
Ext;  5324 

FAX:  (202) -296-7960 
E-mail :  kerchner@rand . org 


Leslie  E  Lan^ella 

HQ  TRADOC,  DCSSA 

Attn:  ATAN-SM 

Fort  Monroe  VA  23651-5143 

OFF  TEL:  (804) -728-5813  DSN:  680-5813 

FAX:  (804) -727-4394 

E-mail :  lampell@monroe-emhl.army.mil 
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PARTICIPANTS  IN  ADVACNCED  DISTRIBUTED  SIMDIiATION  FOR  ANALYSIS  1996  WORKSHOP  as  of  02/10/9 
(Cont . ) 

MAJ  George  Linka 
HQ  ACC/XP-SAS 
204  Dodd  Blvd,  Suite  202 
Langley  AFB  VA  23665-2778 
OFF  TEL:  (804) -764-5757  DSN:  574-5757 
FAX;  (804) -764-7217 

E-mail :  linkag@hqaccxp . langley.af.mil 


COL  James  R  Lingvai 

BMDO/AQM 

The  Pentagon 

Washington  DC  20301-7100 
OFF  TEL:  (703 ) -413- 0300 
FAX:  (703) -413-0337 


LTC  Kemp  L  Littlefield 
HQ  USACOM/JT5 

Joint  Training  Analysis  Simulation  Ctr 
116  Lake  View  Parkway 
Suffolk  VA  23435 

OFF  TEL:  (804) -686-7672  DSN:  564-7672 
FAX:  (804) -686-7501 


Frank  C  Mahncke 

Joint  Warfare  Analysis  Center 

Attn:  J81 

18385  Frontage  Road 

Dahlgren  VA  22448 

OFF  TEL:  (703 ) -663-1872  DSN:  249-1872 
FAX:  (703) -663-1860 


Brian  R  McEnany 
SAIC 

1710  Goodridge  Drive 
Tl-7-2 

McLean  VA  22102 

OFF  TEL:  (703 ) -734-5849 

FAX:  (703) -821-1037 

E-mail :  brian_mcenany®cpqm . saic . com 


Tom  Lucas 
RAND 

PO  Box  2138 
Santa  Monica  CA  90407 
OFF  TEL:  (310) -393-0411 
Ext:  7140 

FAX:  (310) -393-4818 
E-mail :  tom_lucas®rand . org 


LtCol  Douglas  Martin 
Joint  Warfighting  Center 
Fenwick  Road,  Bldg  96 
Fort  Monroe  VA  23651 

OFF  TEL:  (804) -726-6432  DSN:  680-6432 
FAX:  (804) -726-6433 


MAJ  Willie  J  McFadden  II 
HQ  TRADOC 

Fort  Monroe  VA  23651 

OFF  TEL:  (804) -728-5749  DSN:  680-5749 
FAX:  (804) -727-2947 

E-mail :  mcf addew®monroe-emh6 . army.mil 
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PARTICIPANTS  IN  ADVACNCED  DISTRIBUTED  SIMULATION  FOR  ANALYSIS  1996  WORKSHOP  as  of 
(Cont. ) 


CDR  Scot  A  Miller 

CINCPACFLT 

Code  N640,  Box  6 

352  Makalapa  Drive 

Pearl  Harbor  HI  96860-7000 

OFF  TEL:  (808) -474-8444  DSN:  474-8444 

FAX:  (808) -474-8445 

E-mail :  samiller®lono . nose , mil 


Miranda  Moore 

US  Army  Logistics  Integration  Agency 

Attn:  LOIA-AA/M 

54M  Ave . ,  Suite  4 

New  Cumberland  PA  17070-5007 

OFF  TEL:  (717) -770-7600  DSN:  977-7600 

FAX:  (717) -770-6702 


Thomas  K  Neuberger 
CNA  Field  Representative 
JTMDAO  JTF 

1951  Second  Street  SE 
Kirtland  AFB  NM  87117 

OFF  TEL:  (505) -853-0296  DSN:  263-0296 
FAX:  (505) -846-6843 
E-mail :  neubergt@cna . prg 


Roshelle  Orgusaar 
46  Test  Wing/TSWW 
401  W.  Choctawhatchee  Ave 
Suite  265 

Eglin  AFB  FL  32542-5724 

OFF  TEL:  ( 904) -882-9354  DSN:  872-9354 

Ext:  223 

FAX:  (904) -882-9357 

E-mail :  orgusarb@eglin.af.mil 


DR  Louis  R  Moore  III 
RAND 

1700  Main  Street 
PO  Box  2138 

Santa  Monica  CA  90407-2138 
OFF  TEL:  (310) -393-0411 
Ext:  7578 

FAX:  (310) -393-4818 
E-mail:  MOOOOORE@RAND.ORG 


CDR  Lyle  K  Mueller 
Naval  War  College 
Code  33 

686  Cushing  Road 
Newport  RI  02841 

OFF  TEL:  (401) -841-3458  DSN:  948-3458 

FAX:  (401) -841-4628 

E-mail:  lmueller@msis.dmso.mil 


Warren  K  Olson 

Institute  for  Defense  Analyses 
Simulation  Center 
1801  Beauregard  Street 
Alexandria  VA  22311-1772 
OFF  TEL:  (703) -845-2413 
FAX:  (703) -845-6809 
E-mail :  wolson®IDA . ORG 


DR  Dale  K  Pace 

Johns  Hopkins  University/APL 

Analysis  Dept 

11100  Johns  Hopkins  Rd 

Laurel  MD  20723-6099 

OFF  TEL:  (301) -953 -5650 

FAX:  (301) -953-5910 

E-mail:  dale.pace@jhuapl.edu 
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PARTICIPANTS  IN  ADVACNCED  DISTRIBUTED  SIMULATION  FOR  ANALYSIS  1996  WORKSHOP  as  of  02/10/S 
(Cont . ) 


H.  Kent  Pickett 

US  Army  TRADOC  Analysis  Center 

ATTN :  ATRC  -  FM 

255  Sedgwick  Ave 

Fort  Leavenworth  KS  66027-2345 

OFF  TEL:  (913 ) -684-4595  DSN:  552-4595 

FAX:  (913) -684-9232 

E-mail :  pickettk@trac.army.mil 


Tana  P  Reagan 

The  MITRE  Corporation 

MS  Z642 

7525  Colshire  Drive 
McLean  VA  22102 
OFF  TEL:  (703 ) -883-6547 
FAX;  (703) -883-6435 
E-mail :  reagan@mitre . org 


John  D.  Roberts 
The  MITRE  Corporation 
145  Wyckoff  Road 
Eatontown  NJ  07724 
OFF  TEL:  (908 ) -544-6409 
FAX:  (908) -544-8863 
E-mail:  john4@mitre.org 


Michael  D  Roberts 
ASN{RDA)  ACQ  REF 
Crystal  Plaza  5,  Room  924 
Bldg  2211 

Arlington  VA  22202 
OFF  TEL:  (703) -602-5506 
FAX:  (703) -602-5481 

E-mail :  roberts-mike@asnrdad. acq- ref .navy . 


Daniel  M  Rondeau 

Sandia  National  Laboratories 

Department  9803 

MS-0163 

Albuquerque  NM  87185-0163 
OFF  TEL:  (505 ) -845-8420 
FAX:  (505) -844-1196 
E-mail :  dmronde®sandia . gov 


DR  Ernest  A.  Seglie 
DOT&E/OSD 

Pentagon ,  Room  3E3 1 8 

1700  Defense  Pentagon 

Washington  DC  20301-1700 

OFF  TEL:  (703 ) -697-7247  DSN:  227-7247 

FAX:  (703) -693-5248 

E-mail :  seglie_ernest@mail -host. dote. osd. mi 


DR  John  C  Sessler 

BMDO 

AQM 

Pentagon  Room  1E149 

Washington  DC  20301 

OFF  TEL:  (703 ) -413-0314 

FAX:  (703) -602-2496 

E-mail :  sseldon®dmso . dtic . dla . mil 


Dennis  Shea 

Center  for  Naval  Analyses 
4401  Ford  Avenue 
Alexandria  VA  22302-0268 
OFF  TEL:  (703 ) -824-2352 
FAX:  (703) -824-2949 
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PARTICIPANTS  IN  ADVACNCED  DISTRIBUTED  SIMULATION  FOR  ANALYSIS  1996  WORKSHOP  as  of  02/10/96 

(Cont . ) 


LtCol  Robert  S  Sheldon 
AFSAA/SAZ 

The  Pentagon,  Room  1E386 

Washington  DC  20330-1570 

OFF  TEL:  (703) -693-8423  DSN:  223-8423 

FAX:  (703) -697-3441 

E-mail :  sheldon@afsaa.hq.af.mil 


Edward  F  Smith  Jr 

Institute  for  Defense  Analyses 

1801  N.  Beauregard  St 

Alexandria  VA  22311 

OFF  TEL:  (703) -845-6938 

FAX:  (703) -845-2274 

E-mail :  esmith@ida . org 


Richard  C  Smith 

NAWC  WPNS 

Code  52B000E 

521  9th  Street 

Point  Mugu  CA  93042-5001 

OFF  TEL:  (805) -989-8751  DSN:  351-8751 

FAX:  (805) -989-1013 

E-mail :  smithr9@smtpgate . mugu .  navy . mil 


LtCol  Robert  J.  Sirois 
USAF  Air  Warfare  Center 
Det  4,  505th  C2  Eval  Group,  #216 
1655  First  St.,  SE 
Kirtland  AFB  NM  87117-5617 
OFF  TEL:  (505) -846-1472  DSN:  246-1472 
FAX:  (505) -846-1486 


COL  Mark  Smith 

Joint  Advanced  Distributed  Simulation 
1951  2nd  Street  SE 
Kirtland  AFB  NM  87117-5617 
OFF  TEL:  (505) -846-0604  DSN:  246-0604 
FAX:  (505) -846-0603 
E-mail :  smith@jads.kirtland.af.mil 


LtCol  Charles  Snead 

HQ  USAF/XOMT 

1510  Air  Force  Pentagon 

Washington  DC  20330-1510 

OFF  TEL:  (202) -761-5342  DSN:  763-5342 

Ext:  103 

FAX:  (202) -761-5352 
E-mail :  snead@xom .hq.af.mil 


Marchelle  M  Stahl 

Institute  for  Defense  Analyses 

1801  N  Beauregard  St 

Alexandria  VA  22311 

OFF  TEL:  (703) -845-2526 

FAX:  (703) -578-2813 

E-mail :  mstahl@ida . org 


DR  Cyrus  Staniec 
OSD  PA&E 

Theater  Assessments  &  Planning 
The  Pentagon,  Room  2C270 
Washington  DC  20301-1800 
OFF  TEL:  (703) -697-0795 
FAX:  (703) -695-7988 
E-mail :  staniec j@scpael .pae.osd.mil 
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PARTICIPANTS  IN  ADVACNCED  DISTRIBUTED  SIMULATION  FOR  ANALYSIS  1996  WORKSHOP  as  of  02/10/9 
(Cont . ) 


Steven  G  Starner 
SAIC 

1901  N.  Beauregard  Street 

Alexandria  VA  22311 

OFF  TEL:  (703 )- 824 -3418 

FAX:  (703) -379-3778 

E-mail :  sstarner@msis . dmso . mil 


DR  Stuart  H  Starr 
The  MITRE  Corporation 
7525  Colshire  Drive 
MS  W557 

McLean  VA  22102 
OFF  TEL:  (703 ) -883-5494 
FAX:  (703) -883-1373 
E-mail :  starr@mitre . org 


Phillip  G  Thayer 

HQ  ACC/XP-SAS 

204  Dodd  Blvd,  Suite  202 

Langley  AFB  VA  23665-2778 

OFF  TEL:  (804) -764-7227  DSN:  574-7227 

FAX:  (804) -764-7217 

E-mail :  Thayerp®hqaccxp  -  langley.af.mil 


LtCol  Charles  H  Thomas  Jr. 

AFSAA/SAGC 

1570  Air  Force  Pentagon 
Washington  DC  20330-1570 
OFF  TEL:  (703) -697-5616  DSN:  227-5616 
FAX:  (703) -697-1226 
E-mail :  thomasch@af saa .hq.af.mil 


Clayton  J  Thomas  FS 
HQ  USAF  AFSAA/SAN 
1570  Air  Force  Pentagon 
Room  1E387 

Washington  DC  20330-1570 

OFF  TEL:  (703) -697-4300  DSN:  227-4300 

FAX:  (703) -697-3441 

E-mail :  thomasc@afsaa.hq.af.mil 


DR  Jack  Thorpe 

SAIC 

MS  C2 

10260  Campus  Point  Drive 

San  Diego  CA  92121 

OFF  TEL:  (619) -546-6053 

FAX:  (619) -546-6360 

E-mail :  jack_thorpe@cpqm.saic . com 


David  S  Thomen 
SAIC 

8301  Greensboro  Drive,  Suite  460 

McLean  VA  22102 

OFF  TEL:  (703) -749-5131 

FAX:  (703) -847-6328 

E-mail :  dthomen@msis . dmso . mil 


COL  Thomas  J  Verbeck 
Joint  Warfighting  Center 
Fenwick  Road,  Bldg  96 
Fort  Monroe  VA  23651 

OFF  TEL:  (804) -726-6431  DSN:  680-6431 
FAX:  (804) -726-6433 
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PARTICIPANTS  IN  ADVACNCED  DISTRIBUTED  SIMULATION  FOR  ANALYSIS  1996  WORKSHOP  as  of  02/10/96 

(Cont . ) 


Eugene  P  Visco  FS 
SAUS-OR 

102  Pentagon,  Room  1E643 
Washington  DC  20310-0102 
OFF  TEL:  (703) -697-1175  DSN:  227-1175 
FAX:  (703) -697-7748 

E-mail :  visco@pentagon-hqdadss.army.mil 


William  F  Waite 

AEGIS  Research  Corporation 

Suite  200 

6703  Odyssey  Drive 
Huntsville  AL  35806 
OFF  TEL:  (205) -922-0802 
FAX:  (205) -922-0904 
E-mail :  waite@aegisrc . com 


Richard  I  Wiles 

Military  Operations  Research  Society 
101  S  Whiting  Street 
Suite  202 

Alexandria  VA  22304 
OFF  TEL:  (703) -751-7290 
FAX:  (703) -751-8171 
E -mail :  rwiles®dt ic . dla . mil 


DR  Ben  P  Wise 
SAIC 

Suite  130 

20  Burlington  Mall  Road 
Burlington  MA  01803-4126 
OFF  TEL:  (617 ) -221- 7621 
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A2 


AAR 


ACM 

ADS 


ADS-A 


AFTT 


ALMC 


ALSP 


AMG 


APL 


ARPA 


ATD 


AWACS 


AWE 


AWSIM 


BFTT 


BMDO 


BoD 


BOI 


C2 


C3I 


CATIA 


CBS 


CINC 


CJCS 


CJCS 


CM 


CMMS 


COA 


COEA 


CONOPS 


COTS 


DARPA 


DAU 


Anti-Armor  _ 


After  Action  Review 


Air  Combat  Maneuverin 


Advanced  Distributed  Simulation 


Advanced  Distributed  Simulation  for 
Analysis  _ 


Air  Force  Institute  of  Technolo 


Army  Logistics  Management  College 


Aggregate  Level  Simulation  Protocol 


Antenna  Mast  Grou 


Applied  Physics  Laboratory  of  Johns 
Hopkins  University _ 


Advanced  Research  Projects  Agency 


Advanced  Technology  Demonstration 


Airborne  Warning  and  Control  System 


Advanced  Warfighting  Experiment 


Air  Warfare  Simulation 


Battle  Force  Tactical  Trainor 


Ballistic  Missile  Defense  Office 


Board  of  Directors 


Basis  of  Issue  _ 


Command  and  Control 


Command,  Control,  Communications  and 
Intelligence  _ 


Command,  Control,  Communications, 
Computers  and  Intelligence 


Command,  Control,  Communications, 
Computers,  Intelligence,  Surveillance  and 
Reconnaissance  _ 


Computer  Assisted  Three-dimensional 
Interactive  Application 


Coprs  Battle  Simulation  _ 


Commander-in-Chief 


Chairman  of  the  Joint  Chiefs  of  Staff 


Chairman  Joint  Chief  of  Staff  _ 


Configuration  Management 


Conceptual  Models  of  the  Mission  Space 


Course  of  Action  _ 


Cost  and  Operational  Effectiveness 
Analysis  _ 


Concept  of  Operations 


Commercial-Off-The-Shelf 


Defense  Advanced  Research  Projects 
Agency  _ 


Defense  Acquisition  University 


Appendix  C-1 


GLOSSARY  OF  ACRONYMS 


DBMS 


DDR&E 


DEM 


DIAL 


DIS 


DMSO 


DoN 


DSMC 


DT/OT 


EBR 


ECM 


EXCIMS 


GCCS 


HTTL 


HLA 


IADS 

IB 


IDA 


ISR 


rr 


IVIS 


JADS 


JMETL 


JPO 


JS 


JSIMS 


JTF 


JWFC 


K/V 


M&S 


MOE 


MoMs 


MOOTW 


MOP 


MORS 


MORSS 


M-T-M 


NBC 


NDU 


NPS 


NRT 


OLE 


Database  Management  System _ 


Director,  Defense  Research  &  Engineenng 


Distributed  Exercise  Management _ 


Distributed  Intelligent  Architecture  for 
Logistics _ 


Distributed  Interactive  Simulation 


Defense  Modeling  &  Simulation  Office 


Department  of  the  Navy _ 


Defense  Systems  Management  College 


Development  Test/Operational  Test 


Evidence  Based  Research,  Inc. 


Experiment,  Control  and  Monitor _ 


Executive  Committee  for  M&S 


Global  Command  and  Control  System 


Human  in  the  loo 


High  Level  Architecture 


Integrated  Air  Defense  System _ 


Industrial  Base  _ 


Institute  for  Defense  Analyses _ _ 


Intelligence,  Surveillance  and 
Reconnaissance  _ 


Information  Technology  _ _ 


Inter-Vehicular  Information  System _ 


Joint  Air  Defense  System _ 


Joint  Mission  Essential  Task  List 


Joint  Program  Office  _ 


Joint  Staff  _ 


Joint  Simulation  System  _ 


Joint  Task  Force  _ 


Joint  Warfighting  Center _ 


KillerWictim  _ 


Modeling  &  Simulation 


Measures  of  Effectiveness 


Measures  of  Merit  _ 


Military  Operations  Other  Than  War _ 


Measures  of  Performance 


Military  Operations  Research  Societ 


Military  Operations  Research  Society 
Symposium _ 


Model-Test-Model 


Nuclear  Biological  Chemical _ 


National  Defense  Universit 


Naval  Postgraduate  School 


Near  Real  Time 


Obiect  Linking  and  Embeddin 
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ODUSA(OR) 

Office  of  the  Deputy  Undersecretary  of 
the  Army  (Operations  Research) 

OPTEC 

Operational  Test  &  Evaluation  Command 

OR 

Operations  Research 

ORD 

Operational  Requirements  Document 

OSD 

Office  of  the  Secretary  of  Defense 

OSD(PA&E) 

Office  of  the  Secretary  of  Defense 
(Program  Analysis  &  Evaluation) 

PM 

Program  Manager 

PMADS 

Protocol-Mediated  Advanced  Distributed 
Simulation 

RTT 

Receive/Transmit 

RD&A 

Research  Development  and  Acquisition 

RESA 

Research  Evaluation  and  System  Analysis 

RTI 

Run  Time  Infrastructure 

RWR 

Radar  Warning  Receiver 

SAG 

Senior  Advisory  Group 

SAIC 

Science  Applications  International 
Corporation 

SIMNET 

Simulation  Network 

SPI 

Strategic  Perspectives,  Inc. 

STEP 

Standards  for  the  Exchange  of  Product 

TACCSF 

Tactical  Air  Command  and  Control 
Simulation  Facility 

TOR 

Terms  of  Reference 

TRAC 

TRADOC  Analysis  Command 

TRADOC 

Training  &  Doctrine  Command 

USAREUR 

U.S.  Army  Forces:  U.S.  European 

Command 

VR 

Virtual  Reality 

VTC 

Video  Teleconference 

W&A 

Verification,  Validation  and  Accreditation 

w&c 

Verification,  Validation  and  Certification 
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